Model risk management, updated for AI and generative models, so the risk that a model is wrong, misused, or misunderstood is governed across its life.
Executive summary
- Model risk discipline from financial services offers useful concepts for governing AI models, but its regulatory scope must be stated precisely.
- Federal Reserve SR 26-2 and OCC Bulletin 2026-13 replaced the agencies’ 2011 model-risk guidance and expressly exclude generative and agentic AI from their scope.
- Intended-use definition, independent review, monitoring, and change control can still be adapted voluntarily to generative and agentic AI; doing so is a Helixar governance analogy, not a requirement imposed by that US banking guidance.
- Model risk is the risk that a model is wrong, is used outside its valid range, or is misunderstood, and it exists independently of how the model is deployed.
- Generative and agentic models add risks that classic model risk did not face, including confabulation, prompt injection, and opaque behaviour that resists validation.
- Model risk is a lifecycle concern. Validation before deployment is necessary but not sufficient without ongoing monitoring and revalidation on change.
- A validated model can still cause harm through the actions it is allowed to take, so model risk governance must connect to runtime control for agents.
Source basis: Federal Reserve SR 26-2, Model Risk Management; OCC Bulletin 2026-13, Model Risk Management: Revised Comptroller’s Handbook Booklet; ISO/IEC 23894:2023, AI risk management guidance. Full citations and scope notes appear below.
What model risk means for AI
Model risk is the risk that arises from the model itself, distinct from how it is deployed. It is the risk that a model is simply wrong, that it is applied outside the range where it is valid, or that its outputs are misunderstood by the people who rely on them. This framing is powerful because it treats the model as a source of risk in its own right, deserving its own discipline, rather than folding model quality into general software quality where its distinctive failure modes are lost.
For AI, model risk is both familiar and new. It is familiar because organisations have managed the risk of models being wrong for decades, particularly in finance, where models drive lending, pricing, and capital decisions. It is new because AI models, and especially generative ones, fail in ways that older statistical models did not, producing confident falsehoods, behaving differently as inputs shift, and resisting the validation techniques that worked on simpler models. Model risk governance for AI brings the proven discipline to bear on these new failure modes.
The value of a model risk lens is that it focuses attention on the model as a thing to be validated, monitored, and controlled, rather than assumed to work because it performed well in a demonstration. A model that impressed in a pilot may fail in production, on different data, with different users, under different pressure. Model risk governance is the discipline that tests whether a model is actually fit for its intended use and keeps testing it as the world around it changes, which is what prevents a promising model from becoming a costly mistake.
The model risk heritage
Model risk management has a mature financial-services heritage, but the current US supervisory record needs careful boundaries. Federal Reserve SR 26-2 and OCC Bulletin 2026-13 replaced the agencies’ 2011 guidance in 2026. The revised guidance applies a risk-based approach to traditional models and expressly excludes generative and agentic AI. It must therefore not be presented as a direct regulatory mandate for governing those systems.
The earlier 2011 guidance remains historically important because it helped establish practices such as independent challenge, ongoing monitoring, model inventory, and governance across a model portfolio. The current 2026 guidance retains a risk-based model-risk discipline while narrowing and clarifying scope. When this report uses those practices for generative or agentic AI, it does so as a voluntary governance adaptation. Organisations should not cite SR 26-2 or OCC Bulletin 2026-13 as proof that a US banking regulator requires those practices for excluded AI systems.
That distinction does not make the practices less useful. Defining intended use, testing performance and limits, monitoring change, and applying independent challenge are sensible controls for many AI systems. Their authority in this report comes from their governance usefulness and alignment with AI-specific sources such as NIST AI RMF and ISO/IEC 23894, not from extending the legal scope of US model-risk guidance beyond what the agencies wrote.
Model risk within the broader AI risks
Model risk is one part of the broader AI risk picture, and understanding its boundaries prevents both overreach and gaps. Model risk concerns the model being wrong, misused, or misunderstood. It does not, by itself, cover the operational risk of a process failing, the third-party risk of a vendor model, or the runtime risk of an agent taking an unsafe action, though it connects to all of them. Keeping model risk focused on the model, while connecting it to the other risk domains, is what makes it a coherent discipline rather than a catch all.
The relationship between model risk and the risk register is one of specialisation. The risk register records AI risks at the use case level, and model risk is the deeper discipline applied to the model component of those use cases. A use case entry in the register may note that it depends on a model, and the model risk governance provides the validation, monitoring, and review that manage that dependence. The two connect, with model risk feeding the model related entries in the register with evidence that the model is fit for use.
Model risk also connects to but differs from the validation of the whole system. A model may be validated as fit for its intended use and still be part of a system that fails, because the system includes data flows, user behaviour, and downstream actions that model validation does not cover. Model risk governance validates the model; the wider governance validates the use case. Distinguishing these prevents the error of assuming that a validated model means a safe system, which is a dangerous conflation for agentic AI in particular.
The model lifecycle
Model risk is managed across a lifecycle, and describing it as a lifecycle is what turns model risk from a launch check into an ongoing discipline. The lifecycle runs from defining the intended use and validation criteria, through validating the model before deployment, to monitoring its behaviour in production, and controlling change so that a material change sends the model back through validation. Each stage manages a different facet of model risk, and skipping any of them leaves a gap that the others cannot close.
The lifecycle view matters especially for AI because AI models move through their lifecycle faster and less predictably than traditional models. An AI model may be updated by a vendor, retrained on new data, or repurposed for a new use, each of which is a lifecycle event that should trigger a return to validation. An enterprise that manages AI models with a launch and forget approach, validating once and never revisiting, will find its models drifting away from the conditions under which they were validated, carrying risk that its governance no longer sees.
The lifecycle below shows the stages of model risk management. Its value is that it makes explicit that model risk is never finished while a model is in use. Each stage feeds the next, and the change control stage feeds back to validation, so that the lifecycle is a loop that keeps the model within its validated envelope as long as it operates. A model that has left that envelope, through change or drift, without returning to validation, is a model whose risk is no longer governed.
Model risk controls across the lifecycle
Model risk is managed continuously. A material change to model, data, or use sends the model back through validation.
Defining intended use and limits
Model risk governance begins with a clear definition of what the model is for, because a model can only be validated against an intended use, and it can only be misused relative to defined limits. The intended use states the purpose the model is meant to serve, the conditions under which it is valid, the data it is meant to operate on, and the decisions it is meant to inform. Without this definition, validation has no target and misuse has no meaning, because there is nothing the model is supposed to do and therefore nothing it can be said to do wrong.
Defining limits is as important as defining use, because most model failures are failures of use outside the valid range. A model validated for one population may be invalid for another, a model validated for one type of input may fail on another, and a model validated to inform a decision may be dangerous if used to make it. Stating these limits explicitly is what allows the enterprise to catch use outside them, which is one of the most common and preventable sources of model risk, arising not from the model being bad but from it being used where it should not be.
The definition of intended use and limits also becomes the reference for everything that follows. Validation tests against it, monitoring watches for drift away from it, and change control checks whether a change takes the model outside it. A vague or missing definition undermines the whole lifecycle, because there is nothing precise to validate against, monitor against, or control change against. Investing in a clear definition of intended use at the start is what makes the rest of model risk governance possible, and its absence is a common reason model risk governance fails.
Validation before deployment
Validation is the discipline of testing whether a model is fit for its intended use before it is relied upon, and it is the heart of model risk management. Validation tests performance against the intended use, but it goes further, examining robustness to unusual inputs, bias across relevant groups, security against manipulation, and the reasonableness of the model behaviour under conditions it will actually face. Good validation is adversarial in spirit, trying to find where the model fails rather than confirming that it works, because a validation that only seeks confirmation will find it and miss the failures.
For AI models, validation must extend beyond the accuracy metrics that suffice for simpler models. A generative model may score well on a benchmark and still confabulate in production, be manipulable by crafted input, or behave unpredictably as prompts vary. Validation for such models therefore includes testing for these specific failure modes, under realistic and adversarial conditions, rather than relying on aggregate performance. A model that is validated only on average performance is validated against the wrong risks, because the risks that matter for AI often live in the tails and the edge cases.
Validation should be independent of the team that built or selected the model, following the established model risk principle. Independence provides the challenge that self validation cannot, because a team that chose a model has an investment in believing it works. This does not require a large separate function for every model, but it does require that validation be done, or reviewed, by someone with enough distance to be sceptical. The degree of independence should scale with the risk of the model, with high impact models warranting genuinely independent validation.
Risks that generative models add
Generative models introduce risks that classic model risk management did not have to face, and model risk governance for AI must expand to cover them. The most distinctive is confabulation, the production of confident, plausible, but false output, which is dangerous precisely because it is persuasive. A statistical model that is wrong can usually be caught by testing its outputs against ground truth, while a generative model can be wrong in ways that look right, which makes its errors harder to catch and more likely to be trusted. Validation and monitoring must specifically target this.
Prompt injection and manipulation are further generative model risks, where crafted input causes the model to behave in unintended ways, ignoring its instructions or revealing information it should not. This is a security risk that traditional model risk did not consider, because traditional models did not take natural language instructions that could be subverted. Model risk governance for generative models must include testing for injection resilience, treating the model input surface as an attack surface, which is a mindset closer to security than to classic model validation.
Opacity is the third generative model risk, and it complicates the whole discipline. Generative models often cannot explain why they produced a particular output, which makes validation harder, because the validator cannot inspect the reasoning, and makes oversight harder, because the reviewer cannot see the basis of the output. The stack below lists the generative model risks that model risk governance must address. These risks do not replace classic model risk; they are added to it, which is why model risk governance for AI is more demanding than for the statistical models the discipline was built on.
Risks classic model risk did not face
Generative models add these risks to the classic ones. Validation and monitoring must specifically target them.
Monitoring and drift in production
A model that was valid when deployed can become invalid in production, so monitoring is the lifecycle stage that keeps model risk governance connected to reality. The world a model operates in changes: the data it sees shifts, the users interact with it differently, and the context evolves, and any of these can move a model outside the conditions under which it was validated. This is model drift, and monitoring is how the enterprise detects it before it causes harm, rather than discovering it through an incident.
Monitoring for AI models watches more than performance. It watches the distribution of inputs, to detect when the model is being used on data unlike its validation set, the distribution of outputs, to detect behavioural shifts, and the patterns of use, to detect misuse or manipulation. For generative models, monitoring also watches for the specific failure modes: rising confabulation, signs of injection, and outputs that suggest the model is being pushed outside its intended use. Monitoring that watches only aggregate performance will miss the drifts that matter most.
Monitoring must also connect to action, or it is merely observation. When monitoring detects drift, the enterprise needs a response: to investigate, to revalidate, to constrain the model use, or to withdraw it. Monitoring that raises signals no one acts on provides no protection, and the value of monitoring is realised only when it triggers the appropriate lifecycle response. This connects model risk monitoring to the oversight loop, where observation leads to judgement and action, and it is why monitoring should be designed with its responses in mind rather than as data collection for its own sake.
Change control and revalidation
Change is the enemy of validation, because a model that changes is no longer the model that was validated, and change control is the discipline that manages this. When a model is updated, retrained, reconfigured, or applied to a new use, its validation may no longer hold, and change control ensures that such changes trigger a reassessment rather than passing silently. For AI, where models change frequently and sometimes without the enterprise initiating the change, change control is both more important and more difficult than for traditional models.
The particular challenge for AI is that some changes are made by others. A vendor may update a model, or a foundation model may be retrained, changing behaviour the enterprise relied on without the enterprise deciding anything. This means change control for AI must watch for external changes as well as internal ones, connecting to the third-party AI risk discipline of change notification and monitoring. A change control process that manages only the changes the enterprise makes itself will miss the changes made to the models it depends on, which are often the more consequential.
Revalidation is the response to material change, returning the model to the appropriate parts of the lifecycle. Not every change requires full revalidation; a minor change may need only targeted testing, while a major change may require validation from scratch. The discipline is to assess the significance of each change and to revalidate proportionately, so that the model stays within a current validation rather than relying on one that a change has invalidated. A model running on a stale validation, one overtaken by change, is a model whose risk is no longer actually governed.
Independent model review and documentation
Independent review is the model risk principle that provides challenge, and it is what prevents model risk governance from becoming self certification. An independent function, separate from the team that built or selected the model, reviews the validation, challenges the assumptions, and forms its own view of whether the model is fit for use. This independence is uncomfortable by design, and it is precisely its discomfort that makes it valuable, because it surfaces the doubts that an invested team may suppress.
The intensity of independent review should scale with model risk. A low risk model may need only a light independent check, while a high impact model, one driving significant decisions about people or money, warrants genuinely independent and thorough review. This proportionality keeps independent review feasible across an enterprise’s many models while concentrating rigorous challenge where the consequences of a model failure would be greatest. Applying uniform review either overburdens low risk models or underserves high risk ones.
Documentation is what makes model risk governance auditable and what carries knowledge across the model life. Model risk documentation records the intended use, the validation, the monitoring, the changes, and the reviews, so that anyone relying on the model can understand its risk and anyone auditing it can test whether the discipline was applied. The matrix below shows the documentation model risk governance should retain. Good documentation is not bureaucracy; it is what allows a model to be understood and challenged by people other than those who built it, which is the foundation of independent governance.
What model risk governance should document
Documentation makes model risk auditable and carries knowledge across the model life, so the model can be understood by people other than its builders.
Model risk for agents
Model risk governance is necessary for agentic AI but not sufficient, and understanding why is essential to governing agents safely. A model can be thoroughly validated, monitored, and reviewed, and still cause harm through the actions an agent built on it is allowed to take. Validation tells the enterprise the model is fit for its intended use; it does not tell the enterprise that the agent using the model is safe, because the agent risk lies in what it can do, not only in whether the model is correct. A validated model that can take unsafe actions is a governed model in an ungoverned system.
This means model risk governance for agents must connect to runtime control. The model risk discipline validates the model and monitors its behaviour, and the runtime governance bounds and records what the agent does with it. The two together govern the agent: the model is fit for use, and its use is constrained to safe actions. Relying on model validation alone to govern an agent is the conflation of a validated model with a safe system, and it is a dangerous mistake for agentic AI, where the gap between the two is exactly where the risk lives.
The practical implication is that an enterprise deploying agents should not treat a strong model validation as sufficient assurance. It should ask, separately, what the agent can do, whether those actions are bounded, and whether the boundaries are enforced and recorded. Model risk governance answers whether the model is right; runtime governance answers whether its use is safe. An enterprise that has the first without the second has validated the engine without governing the vehicle, which is why model risk governance and runtime control are complementary rather than alternative.
Common model risk failures
The most common model risk failure is the launch and forget approach, where a model is validated once and never revisited, so that drift and change silently invalidate the validation. The remedy is the lifecycle discipline with monitoring and change triggers, which keeps the model within a current validation. The second common failure is validation that only seeks to confirm, testing the model until it passes rather than trying to find where it fails, which produces a validation that misses the real risks. The remedy is adversarial validation that hunts for failure.
A third failure is treating a validated model as a safe system, particularly for agents, which conflates the correctness of the model with the safety of its use. The remedy is to connect model risk governance to runtime control, governing what the agent does as well as validating the model. A fourth is the absence of independent review, where a team validates its own model and finds it good, missing the challenge that independence provides. The remedy is independent review proportionate to the model risk.
Each of these failures reflects a partial adoption of model risk discipline, taking the validation gate but not the lifecycle, the testing but not the independence, or the model focus but not its connection to the system. The remedy in each case is to adopt the discipline fully: define intended use, validate adversarially and independently, monitor for drift, control change, and connect model risk to the wider governance of the use case. The heritage of model risk management shows that this discipline works; the task for AI is to apply it completely rather than partially.
Conclusion: the Helixar perspective
The Helixar research perspective is that model validation is necessary but not sufficient for agentic AI. A model can be validated to the highest standard and still cause harm through the actions it is allowed to take, because the agent risk lies in its authority, not only in the correctness of its model. Model risk governance answers whether the model is right, and it must connect to runtime governance, which answers whether the use of the model is safe. The two together, not either alone, govern an agent.
This is why operational policy governance complements model risk governance rather than duplicating it. Model risk governance validates and monitors the model, applying a proven discipline to a new class of model. Operational policy governance bounds and records what the agent does with the model at runtime. An enterprise that has both governs the engine and the vehicle; one that has only model validation has confirmed the model is correct while leaving what it does unconstrained, which for an agent is the more important question.
Read alongside the risk register and operational risk reports, this report shows how the enterprise governs the model as a source of risk: by defining its intended use, validating it adversarially and independently, monitoring it for drift, controlling change, and connecting model risk to the safety of the system that uses it. Model risk has a mature heritage; the task for AI is to apply it fully and extend it for generative and agentic behaviour. For the whole discipline these reports support, the Enterprise AI Governance Framework is the anchor.
Enterprise checklist
- Define the intended use, valid conditions, and limits of every model.
- Validate performance, bias, security, and robustness before deployment, adversarially.
- Test generative models for confabulation, prompt injection, and opaque behaviour.
- Monitor for drift in inputs, outputs, and use, and connect monitoring to action.
- Control change and revalidate proportionately, watching for vendor model updates.
- Apply independent review scaled to the model risk, and document the whole lifecycle.
- Connect model risk to runtime control, because a validated model can still act unsafely.
Frequently asked questions
Does financial model risk guidance apply to generative AI?
Is model validation enough for agents?
What is model drift and why does it matter?
Why does validation need to be independent?
What generative model risks must validation cover?
Method and source use
This report is a Helixar synthesis of the cited public standards and guidance. Named sources are linked at first mention and listed below. Unless a cited source is identified, maturity levels, diagrams, allocations, scores, and operating models are illustrative Helixar reference models, not survey findings or legal requirements. Organisations should verify current obligations with the authoritative source and qualified advisers.
References
- Federal Reserve SR 26-2, Model Risk Management
- OCC Bulletin 2026-13, Model Risk Management: Revised Comptroller’s Handbook Booklet
- ISO/IEC 23894:2023, AI risk management guidance
- NIST AI RMF Generative AI Profile (NIST.AI.600-1)
- OWASP Top 10 for LLM Applications (2025)
- Helixar research: Enterprise AI Risk Register Framework
- Helixar research: Enterprise AI Governance Framework