AI as an operational risk, not only a model risk, brought into the resilience discipline that already governs critical processes.
Executive summary
- Once AI is embedded in a business process, it becomes an operational risk, because if the AI degrades, behaves unexpectedly, or is disrupted, the process it supports can fail.
- Operational resilience frameworks give enterprises a ready and proven structure for AI dependency and disruption, so AI operational risk need not be governed from scratch.
- AI adds new operational failure modes, including unsafe tool use, runaway cost, and cascading errors that develop faster than a human can intervene.
- Incident response for AI must extend to these new failure modes and include the ability to contain, meaning to slow, pause, or stop an AI quickly when its behaviour is unsafe.
- Containment is an operational control, not a safety afterthought, and for agents it is the difference between an incident that is stopped and one that runs to its conclusion.
Source basis: APRA Prudential Standard CPS 230 Operational Risk Management; Basel Committee, Principles for Operational Resilience; ISO 22301:2019, Business continuity management systems. Full citations and scope notes appear below.
AI as an operational risk
Model risk concerns the model being wrong, but operational risk concerns the process failing, and as AI becomes embedded in how the enterprise operates, it becomes a source of operational risk in its own right. When a business process depends on an AI system, the reliability of that process now depends on the reliability of the AI, and a failure, degradation, or disruption of the AI becomes a failure of the process. This is a different risk from the model being inaccurate; it is the risk that the process the model supports stops working.
This operational dimension is easy to underweight, because attention naturally focuses on whether the AI is correct rather than on what happens to the process if the AI is unavailable or behaves unexpectedly. But an AI that is usually accurate can still cause an operational failure if it goes down at a critical moment, if it suddenly behaves differently after an update, or if it generates work faster than downstream systems can handle. Operational risk management focuses on these process level failures, which model risk management does not address.
Bringing AI into operational risk management means treating an AI dependency the way the enterprise treats any other critical dependency: understanding what depends on it, how it could fail, what the impact would be, and how the enterprise would keep operating if it did. This is a mature discipline in most enterprises, applied to systems, suppliers, and infrastructure, and extending it to AI is more a matter of applying existing practice than inventing new. The novelty is in the AI specific failure modes, which the existing discipline must expand to cover.
When AI becomes operational risk
AI becomes an operational risk at the point where a process depends on it, and identifying that point is the first step in managing it. A model used to draft suggestions that a human may or may not use creates little operational dependency, because the process can proceed without it. A model wired into a process such that the process cannot complete without it, or an agent that performs a step no human backs up, creates a genuine operational dependency, where a failure of the AI is a failure of the process. The degree of operational risk tracks the degree of dependency.
The dependency often deepens quietly, which is what makes it dangerous. An AI capability introduced as an assistant, which the process could function without, can become load bearing over time as people come to rely on it and the manual fallback atrophies. What began as a convenience becomes a dependency, and the process that could once operate without the AI can no longer do so, often without anyone deciding that this should be the case. Operational risk management watches for this deepening dependency, because it changes the risk without any explicit decision.
Autonomy sharply increases operational dependency, because an autonomous agent performs steps that no human is positioned to perform if it fails. When a process relies on a human who uses AI, the human remains as a fallback if the AI fails. When a process relies on an agent that performs a step autonomously, there may be no human ready to step in, so an agent failure is more directly a process failure. This is why agentic AI raises operational risk more than assistive AI, and why the operational resilience of agent dependent processes deserves particular attention.
Operational resilience frameworks
Enterprises do not need to invent operational risk management for AI, because mature frameworks already exist and AI dependencies fit within them. In Australia, the prudential standard APRA CPS 230 requires regulated entities to manage operational risk and maintain the resilience of critical operations, including through material service providers, and AI dependencies fit naturally into this. Internationally, the Basel Committee principles for operational resilience set out how institutions should identify critical operations, map dependencies, set tolerances for disruption, and test their ability to keep operating.
These frameworks share a common structure that applies directly to AI. They require an enterprise to identify its critical operations, understand what those operations depend on, set a tolerance for how much disruption is acceptable, and test that it can stay within that tolerance when something fails. Applying this structure to AI means identifying which critical operations depend on AI, mapping those dependencies, setting tolerances for AI disruption, and testing resilience to AI failure. The framework provides the structure; AI provides a new class of dependency to manage within it.
Business continuity standards such as ISO 22301 provide a further foundation, extending the discipline to planning for and recovering from disruption. An AI dependency should be part of business continuity planning, with a defined response for the case where the AI is unavailable or unusable. Using these established frameworks avoids building a parallel AI resilience scheme and connects AI operational risk to the resilience governance the enterprise already operates, which is where it belongs, because AI disruption is a form of operational disruption rather than a separate category.
Critical processes and AI dependencies
Managing AI operational risk begins with knowing which critical processes depend on AI, which requires mapping AI dependencies against the enterprise critical operations. This is often revealing, because AI dependencies accumulate without a clear record, and an enterprise may not realise how many of its important processes now rely on AI until it maps them. The mapping identifies where an AI failure would disrupt a critical operation, which is where operational risk management should concentrate.
Dependency mapping should capture not only the direct dependencies but the chains, because a process may depend on an AI that itself depends on a vendor model, which depends on infrastructure. A disruption anywhere in this chain can reach the critical process, so understanding the full dependency chain is necessary to understand the real resilience of the process. A mapping that captures only the direct AI dependency, without the chain behind it, understates the risk by ignoring the deeper dependencies that could equally cause a failure.
The mapping also reveals concentration and single points of failure, where several critical processes depend on the same AI or the same provider. Such concentration means a single AI failure could disrupt multiple critical operations at once, which is a more serious risk than any individual dependency. Identifying these shared dependencies is what allows the enterprise to manage them deliberately, whether by strengthening the shared dependency, providing alternatives, or accepting the concentration knowingly. Concentration that is mapped can be managed; concentration that is unmapped is a hidden systemic risk.
Tolerance for disruption
Operational resilience is built around tolerances for disruption, which state how much disruption to a critical operation is acceptable before it causes unacceptable harm. Setting these tolerances for AI dependent operations forces the enterprise to decide, in advance, how long a critical process could tolerate an AI being unavailable or degraded, and what the enterprise would do as that limit approached. This is a more useful discipline than assuming AI will always be available, because it prepares the enterprise for the failure rather than being surprised by it.
Tolerances should be set from the impact of disruption, not from the expected reliability of the AI. It is tempting to reason that because an AI is reliable, disruption is unlikely, and therefore tolerances do not matter, but this confuses likelihood with impact. A rare disruption of a critical operation can still be catastrophic, and the tolerance exists precisely for the rare case. Setting tolerances from impact ensures the enterprise is prepared for the disruption that matters, however unlikely, rather than being caught out because it judged the disruption improbable.
Tolerances also drive the design of fallbacks and alternatives, because meeting a tolerance requires the ability to keep operating through a disruption. If a critical process must tolerate no more than a short AI outage, the enterprise needs a way to keep the process running during a longer one, whether through a manual fallback, an alternative system, or a degraded mode. The tolerance defines the requirement, and the fallback meets it. An enterprise that sets tolerances without building the fallbacks to meet them has stated an intention it cannot honour when the disruption comes.
AI specific operational failure modes
AI introduces operational failure modes that traditional operational risk did not have to anticipate, and operational risk management for AI must expand to cover them. Unsafe tool use is one, where an agent invokes a capability in a harmful way, causing an operational impact through an action rather than an output. Runaway cost and scale is another, where an agent generates work, requests, or spending faster than expected, producing a financial or capacity impact at machine speed. Cascading errors are a third, where an AI error propagates through connected systems before anyone notices.
These failure modes share a characteristic that makes them operationally dangerous: they can develop faster than human intervention. A human process fails at human speed, giving people time to notice and respond, but an agent can take many actions, generate large costs, or propagate an error across systems in the time a human takes to notice something is wrong. This speed is what makes AI operational failures distinctive, and it is why the operational controls for AI must include the ability to detect and respond quickly, rather than relying on periodic human review.
The matrix later in this report sets out the operational risk domains where these failures concentrate. The key point is that AI operational failure modes are not exotic; they are the predictable consequences of embedding fast, autonomous systems in business processes. An enterprise that anticipates unsafe tool use, runaway cost, and cascading errors can build controls against them, while one that governs AI operational risk with a framework designed for slower, human paced failures will be surprised by how quickly an AI operational failure can escalate beyond its ability to respond.
Incident response for AI
Incident response for AI extends the enterprise incident management to the specific ways AI fails, and it must be fast enough to matter given the speed of AI failures. An AI incident may involve incorrect output that influenced a decision, unauthorised data exposure, a harmful recommendation, unsafe tool use, a cost runaway, or a cascade of errors. The incident process should detect these, contain them, remediate the harm, and feed the lesson back into governance, the same shape as any incident response but tuned to AI failure modes.
The distinctive requirement for AI incident response is speed, because an AI incident can escalate quickly. The flow below shows the incident response cycle: detect, contain, remediate, and learn. For AI, the containment step is especially critical and especially time sensitive, because an agent that is behaving unsafely will keep acting until it is stopped, and every moment of delay allows more impact. Incident response for AI that can detect a problem but cannot contain it quickly is response that arrives too late to prevent the harm.
The learning step closes the loop and is what makes incident response improve governance rather than merely handle failures. Each AI incident should feed back into the risk register, the model risk governance, the oversight design, and the policy, so that the enterprise gets safer as it experiences and responds to failure. An enterprise that handles each AI incident in isolation, without feeding the lesson back, will see the same failure modes recur, while one that learns from its incidents progressively closes the gaps that allowed them. Incident response is not only about recovery; it is about learning.
Detect, contain, remediate, learn
AI incidents can escalate quickly, so containment must be fast. The learning step feeds the lesson back into governance.
Identify the incident quickly, given AI speed.
Slow, pause, or stop the AI to limit impact.
Correct the harm and restore the process.
Feed the lesson into risk, model, and policy.
Containment and kill controls
The ability to contain an AI, to slow, pause, or stop it quickly when its behaviour is unsafe, is the operational control that agentic AI makes essential and that many enterprises lack. Containment is not a safety feature bolted onto the system; it is a core operational control, the equivalent of an emergency stop on a machine. An enterprise that can watch an agent but cannot stop it has visibility without control, which means it can see an operational failure developing and do nothing to arrest it in time.
Containment must be designed and, crucially, tested, because a containment capability that has never been exercised is a claim rather than a control. The enterprise should be able to demonstrate that it can actually stop an agent, quickly, in the ways that matter: halting a specific action, pausing an agent, or shutting down a class of activity. Testing containment reveals whether it works, and it is far better to discover a containment failure in a test than during an incident, when the failure to stop an unsafe agent could be costly.
Containment also needs to be proportionate and precise, so that stopping an unsafe activity does not needlessly disrupt safe ones. A blunt containment that can only shut everything down is better than none, but a precise containment that can stop a specific unsafe action while allowing others to continue is far more useful, because it limits the operational impact of the containment itself. This precision comes from governing AI at the level of individual actions, which is the runtime governance that operational policy governance provides, and it is what makes containment a scalpel rather than a sledgehammer.
The operational risk domains
AI operational risk concentrates in a small number of domains, each of which needs a control focus and evidence that the control operates. Process dependency is the domain of critical processes relying on AI, managed through dependency mapping, tolerances, and contingency. Resilience is the domain of keeping operating through disruption, managed through fallbacks and testing. Incident response is the domain of detecting and containing AI failures. And cost and scale is the domain of the runaway financial and capacity risks that autonomous systems create.
Setting out the domains explicitly helps an enterprise see where its AI operational risk management is complete and where it is thin. Many enterprises have some incident response but little containment, or have mapped dependencies but not set tolerances, or have not considered the cost and scale domain at all. The matrix below shows the domains, their operational concern, and the evidence that the control operates. Laying them out this way turns a vague sense that AI creates operational risk into a specific set of domains to govern, each with a defined control.
The domains also connect to the wider operational risk framework, so that AI operational risk is managed alongside other operational risk rather than separately. Process dependency and resilience connect to the enterprise operational resilience programme, incident response connects to the enterprise incident management, and cost and scale connect to financial controls. This connection is what makes AI operational risk a normal part of operational risk management rather than a special case, which is where it should sit, because AI disruption is operational disruption.
Where AI operational risk concentrates
Each domain needs a control focus and evidence that the control operates when it matters.
Resilience testing
Resilience is a claim until it is tested, and testing is what turns a resilience plan into a demonstrated capability. Resilience testing for AI simulates the failure of an AI dependency and verifies that the enterprise can keep its critical operations running within tolerance. This might mean testing that a process can proceed with the AI unavailable, that a fallback works, or that a degraded mode is acceptable. Testing reveals whether the resilience the enterprise believes it has is real, and it frequently reveals gaps that were invisible on paper.
Testing should include the AI specific failure modes, not only simple unavailability. It is one thing to test that a process copes when an AI is down, and another to test that it copes when the AI is available but behaving unexpectedly, generating errors, or running away in cost. These behavioural failures are harder to test but often more likely than a clean outage, and a resilience programme that tests only for availability, not for misbehaviour, is testing for the easier failure while leaving the harder one unverified.
The results of resilience testing feed back into the tolerances, the fallbacks, and the dependency management, closing the loop. A test that reveals a process cannot meet its tolerance shows that either the tolerance is wrong or the fallback is inadequate, and either way the enterprise learns something it needs to know before a real disruption. Resilience testing that is done and then ignored provides false comfort; resilience testing that feeds its findings back into the resilience design is what progressively makes the enterprise genuinely able to keep operating through AI disruption.
Cost and scale risk
Cost and scale risk is the operational failure mode most specific to autonomous AI, and it deserves particular attention because it can escalate faster than almost any other. An agent that is misconfigured, manipulated, or simply behaving unexpectedly can generate requests, actions, or spending at machine speed, producing a large financial or capacity impact before anyone notices. Unlike a human process, which fails at a rate a human can observe, an agent cost runaway can accumulate a significant cost in the time it takes someone to realise something is wrong.
Managing cost and scale risk requires controls that operate at machine speed, because human oversight cannot keep pace. Rate limits cap how quickly an agent can act, budgets cap how much it can spend, and monitoring with automated alerting detects abnormal rates of activity. These controls are operational rather than model related, and they are essential for any agent that can incur cost or generate scale. The bars below show, illustratively, how the risk of runaway impact rises with agent autonomy and the absence of these controls.
The deeper point is that cost and scale risk illustrates why AI operational risk needs machine speed controls generally. The failure modes that matter most for autonomous AI, cost runaway, cascading errors, unsafe action, all develop faster than human intervention, so the controls against them must operate automatically. An enterprise that relies on human oversight to catch these failures will catch them too late, which is why operational risk management for autonomous AI leans heavily on automated limits, monitoring, and containment rather than on human vigilance alone.
Illustrative runaway risk by autonomy and controls
How the risk of runaway cost and scale rises with agent autonomy and the absence of machine speed controls.
Conclusion: the Helixar perspective
The Helixar research perspective is that operational resilience for AI needs a containment point. The ability to slow, pause, or stop an AI when its behaviour is unsafe is an operational control, not a safety afterthought, and for agents it is the difference between an incident that is arrested and one that runs to its conclusion. When containment operates at the level of individual actions and is recorded, the enterprise can stop an unsafe activity precisely, limit the impact, and show what happened, which is exactly what an operational resilience review looks for.
This is where operational policy governance provides the operational controls that autonomous AI requires: the containment to stop an unsafe agent, the runtime enforcement to bound its actions, and the records that show the controls operated. These are operational controls, operating at machine speed, because the failures they guard against develop faster than human intervention. An enterprise that has them can keep operating through AI disruption; one that relies on human vigilance alone will find that AI operational failures outrun its ability to respond.
Read alongside the risk register and model risk reports, this report shows how the enterprise governs AI as an operational risk: by mapping dependencies, setting tolerances, testing resilience, responding fast to incidents, and containing autonomous systems that can fail at machine speed. AI operational risk fits within the resilience frameworks the enterprise already uses, extended for the failure modes AI adds. For the whole discipline these reports support, the Enterprise AI Governance Framework is the anchor.
Enterprise checklist
- Identify which critical processes depend on AI, and map the full dependency chain.
- Set tolerances for AI disruption from impact, not from expected reliability.
- Build and test fallbacks that meet the tolerances you set.
- Extend incident response to AI failure modes, and make containment fast.
- Provide containment and kill controls, and test that they actually work.
- Add machine speed controls for cost and scale: rate limits, budgets, and alerting.
- Feed every AI incident and resilience test back into governance.
Frequently asked questions
How is AI operational risk different from model risk?
Which frameworks apply to AI operational risk?
Why is containment so important for AI?
What is cost and scale risk?
How do we know our AI resilience is real?
Method and source use
This report is a Helixar synthesis of the cited public standards and guidance. Named sources are linked at first mention and listed below. Unless a cited source is identified, maturity levels, diagrams, allocations, scores, and operating models are illustrative Helixar reference models, not survey findings or legal requirements. Organisations should verify current obligations with the authoritative source and qualified advisers.
References
- APRA Prudential Standard CPS 230 Operational Risk Management
- Basel Committee, Principles for Operational Resilience
- ISO 22301:2019, Business continuity management systems
- ISO 31000:2018, Risk management guidelines
- NIST AI Risk Management Framework (AI RMF 1.0)
- Helixar research: APRA CPS 230 and AI Governance
- Helixar research: Enterprise AI Governance Framework