Safeguarding Operational Resilience in the Era of AI-assisted Coding

The Cambridge Centre for Alternative Finance (CCAF) 2026 Global AI in Financial Services Report, conducted among regulators, traditional FIs and AI vendors, has revealed a consensus forming around the industry's understanding of artificial intelligence (AI) risks (University of Cambridge, 2026). The report cites 'software engineering' as 'the financial industry's most mature AI application' and defines it as a primary direction of cyber risk transmission. Fifty-one per cent of respondents across the three sectors cite the 'loss of human oversight' due to automation (and the proliferation of 'collective forgetting') as the third-highest AI risk, preceded by 'data privacy & protection risks' and 'model hallucinations & unreliable outputs' as the respective top two risks identified by all stakeholder groups. In software engineering, as the report remarks, the 'unprecedented volume and velocity of AI-generated code make traditional manual reviews increasingly ineffective'. So the question is: how does AI-assisted code really fit with the existing infrastructure, and what is a sustainable approach to achieving its production readiness?

This discussion is focused on the use of AI tools for the purpose of increasing developer output, ultimately to increase delivery speed. It is important to differentiate between two types of coding that resort to AI tools:

• Vibe coding – the process of generating code by a person who has no expertise in assessing the syntactic or business-logic-related quality of that code and, consequently, no presumed accountability for it. It remains outside the scope of this article.
• AI-assisted coding – a process where professional developers use AI tools to automate routine coding tasks – or even the generation of the bulk of their code – whilst retaining the professional ability to review such code (ideally, also having a contractual obligation to do so). How fully that ability is exercised within the constraints of increasing workloads and shrinking deadlines remains to be seen. However, we focus on this type of developer as the only one relevant in the regulated financial systems context.

Not all that 'shines' is code

Another recent report conducted among 200 enterprise technology leaders – State of Code Abundance 2026 – reveals a confidence level of 92% in relation to AI-generated code, yet a staggering 81% of respondents admit experiencing an increase in production issues linked to AI-generated code (CloudBees, 2026). They also admit struggling to see clear production gains attributed to incorporating AI into their delivery pipelines. Other reported challenges include 'token anxiety', difficulties in forecasting AI consumption costs, as well as rising 'CI/CD, testing, security scanning' budgets. Nearly two-thirds (64%) of respondents describe AI as 'widely adopted' or 'fully integrated'. Yet very few – 12% – report having a dedicated AI governance team.

A developer does have the potential to see their productivity increase with the help of AI tools. However, in the regulated space, that improvement should not exceed the speed at which the developer can consciously review every element of the code that has been artificially generated, to fully stand behind it. They should also be aware of the unconscious biases that can affect human 'sensemaking' processes, including code review: automation bias, confirmation bias, inattentional blindness (Bach, J., Bolton, M., 2025), and cognitive surrender, just to name a few. They should also make a deliberate effort to transition into a new role – reviewer instead of hands-on practitioner – while accepting the threat of gradual deskilling (Jordanovski, M., 2025).

Subsequently, enterprises may eventually get to a level of automation where they lose the 'institutional memory and capabilities to execute processes manually if required, particularly when crises hit' (hence the term 'collective forgetting') (University of Cambridge, 2026).

Careful what you automate for…

More than half (52%) of those surveyed by CloudBees reported an AI-related uptick in software development output (with 61% of the surveyed engineering organisations' code reportedly being generated or assisted by AI). When asked about the 'primary bottleneck in their delivery sites today', 57% of the technology leaders point to 'reviewing', 'testing' and 'deploying' code, while only 35% indicate 'writing' code. Failures happening post-deployment are attributed to the validation process not 'keeping pace' with what AI is producing (CloudBees, 2026).

This cannot but take a toll on teams tasked solely with integration, acceptance and system-level testing (be it provided independently or not). It contributes to an already erroneous image of testing as an impediment to streamlined delivery, instead of being thought of as 'the final frontier' and an enabler of cleaner architecture, more informed decisions and, ultimately, more reliable technology. These benefits, however, come as a result of a continuous, collaborative, inter-stakeholder effort – something that AI takes away completely. Instead, it perpetuates one-sided decisions and code implemented based on the image of completeness imposed on AI models externally by the training data. Introduced at the enterprise level, this 'alien' logic can accumulate technical debt and create levels of complexity that could be impossible to untangle.

System failures and external events are reported by the Joint Committee of the European Supervisory Authorities as the predominant driver of major information and communication technology (ICT) incidents occurring in 2025 (ESA, 2026). Their impact on clients, transactions and financial counterparties has been assessed as 'limited'. However, combined with 'collective forgetting', unchecked and overwhelming complexity can quickly lead to increased and more impactful ICT disruptions, as well as a dramatically weakened ability to bring systems back online within reasonable timeframes.

Short-term gains: easy come, easy go

Developers – who have traditionally relied on formal specifications to produce code – may now also receive much less formal 'cues' to do their work. In environments where AI-assisted coding is heavily relied upon, entire specification documents may gradually be replaced with less formal, natural-language requests to save time. The amount of tacit organisational and domain knowledge developers apply while working on any given task – even one that has a clearly articulated specification – is quite high. With a prompt-like feature request and no clear guardrails on implementation, the outcome could range from naive to highly overfitted to a specific system or organisation, rather than to the one that ends up with the code.

In the absence of formal specifications, testers are also left without test oracles – reliable evidence of how a system is officially expected to behave. Where the amount of code multiplies exponentially, it seems fair that so should the number of testers, or at least their capacity to test efficiently and effectively. But inherently unreliable code enables an additional layer of verification: fundamentally flawed code logic, irrelevant decisions and dependencies, non-existent classes, and excessive complexity – issues that would rarely have left the developer remit until now. By delegating coding to AI in a manner that is not adapted to producing high-quality code tailored to the project and business goals, developers may essentially end up delegating their job to testers, and then conveniently calling the next people down the line the 'bottleneck'.

We are already witnessing highly disjointed versions of accountability measures when it comes to documented service disruptions. According to the CloudBees report, anyone from an individual developer who committed the code all the way through to a CTO could be held accountable should a production-disruptive incident occur. Termed a 'problem without ownership' (CloudBees), the phenomenon is likely a combination of problems potentially spanning functional and non-functional areas, including 'security vulnerabilities, and compliance violations that reach production because governance and validation have not scaled with output' (Claburn, 2026).

Guardrails for sustainable growth and continued resilience

The bottom line is that introducing AI tools as an isolated measure at the heart of enterprise software delivery will likely have a negative impact on long-term resilience. To prevent the situation from escalating into a worst-case scenario – which is a question of 'when', not 'if' – organisations should define and consolidate all business and operational aspects related to AI use into a coherent framework that evolves consistently across its components.

This concerns every aspect of delivery and includes governance, strategy, security, operational resilience, team skills and composition, automation, and testing. A comprehensive test strategy developed early on may significantly improve the outcomes of AI integration in regulation-first platforms.

The testing efforts should not only be concentrated on functional correctness, but must prioritise setting the guardrails for determinism, resilience, security, and auditability. Testing itself must transition from a finite process to a continuous, multi-layered framework embedded across the delivery lifecycle. Knowledge gaps should be proactively identified and closed enterprise-wide to accommodate the technology transition.

As a specialist software testing company, we partner with organisations to build AI testing strategies that keep pace with the speed and scale of AI-enabled development. We help teams move fast without compromising on security, reliability, or trust – and ensure that what gets deployed does not just qualify as 'production-ready', but is built to endure.