Editor’s Note — This article is sponsored by C&R Software. As with all sponsored content in Fintech Takes, this article was written, edited, and published by me, Alex Johnson. I hope you enjoy it!
There are two ways organizations have learned to govern risk.
The first was built for software; the second was built for people. Neither was built for AI agents.
Large language models (LLMs) and the agentic AI systems built around them are categorically new.
Not new in the way that every technology is new. New in the sense that they work fundamentally differently than the analytic models and software we already know how to govern.
They’re probabilistic systems. They don’t follow rules so much as they interpret them (and that interpretation can be different every time).
This is what makes AI so powerful, but it’s also why you can’t govern AI agents the same way you govern software or people.
We can’t just slot AI agents into the compliance structures we already have. To understand why, it helps to look at what those structures were built for.
Software
Before AI, banks relied exclusively on traditional machine learning models to make decisions around credit scoring, fraud detection, risk assessment, and the like. For the past 15 years, SR 11-7 has been the U.S. governance standard that has defined how to manage them.
Its logic rests on one core concept: replicability.
One team designs and builds the model. A completely separate team validates it; feeding the same data in and checking that the same outputs come out, every time.The entire framework rests on that assumption. You can test it before deployment, audit it after, and hold it accountable because it behaves predictably.
Deterministic systems always produce the same result from the same inputs. Replicability is possible which means accountability is possible.
Humans
Governance frameworks built for humans start from the opposite assumption: someone will make a mistake. They’ll click the phishing link, or rationalize making an exception just this one time.
To quote the psychologist James Reason, “We cannot change the human condition, but we can change the conditions under which humans work.”
The solution is to design systems that survive imperfection.
Reason called his framework the Swiss Cheese Model of Accident Causation. Every individual slice of Swiss cheese has holes in it, but if you layer enough slices, the holes cancel each other out from a risk perspective.
The Bank for International Settlements (BIS), the international organization that sets standards for banking supervision, applies a similar logic to operational risk through its Basel Committee on Banking Supervision. Their updated 2021 framework defines operational risk as “the risk of loss resulting from inadequate or failed internal processes, people and systems or from external events.”
Their answer was three slices of Swiss cheese (otherwise known as the three lines of defense: business unit management, independent risk management, internal audit), each fallible on its own but stronger together as a stack.
Where Does AI Fit?
LLMs and agentic AI don’t fit cleanly on either the software or human side of the risk management continuum.
As we’ve seen, AI has elements of deterministic software. You can give AI rules, run it continuously at a scale no human team could match, but it also doesn’t behave like software.
ISO 42001, the first international standard for AI management systems directly acknowledges this:
AI systems that perform continuous learning change their behaviour during use.
AI systems that perform continuous learning change their behaviour during use.
They require special consideration to ensure their responsible use continues with changing behaviour.
Are AI systems more like humans?
Kind of. They’re not unlike human interns who find solutions you didn’t anticipate and make judgment calls you didn’t authorize. However, that comparison is lacking because the speed and capability of modern AI systems far outsrips what any human (or team of humans) is capable of.
The truth is that AI sits somewhere in the middle of the continuum. Indeed, regulators have even begun to acknowledge this by descoping LLMs and agentic AI systems from traditional model risk management guidance.
So what could a philosophy for governing AI agents look like? And how would that philosophy translate into technical requirements?
I’m so glad you asked!
A New Philosophy
The solution would need to be designed with multiple layers of safeguards to ensure a baseline level of safety despite LLMs’ inherent unpredictability. Such safeguards might include:
- Human-in-the-loop. The AI only presents suggestions while humans make decisions. This preserves accountability in a system that can’t be held accountable on its own.
- No autonomous actions. Humans retain full control and responsibility, overriding AI’s sycophantic desire to please and occasional impulse to destroy. AI doesn’t directly communicate with customers or take independent actions.
- Policy-driven responses. All AI-generated suggestions are strictly based on company-approved documentation, with no inputs from the internet or other outside sources.
- Audit trail. Every interaction is logged for compliance review/traceability. You can’t govern what you can’t see.
These four guardrails should connect through a single pane of glass; that is, all data and decisions flow through one system.
If a bank doesn’t have a clear and holistic picture of its customers, how will it ever govern AI agents operating across those same divisions?
Why Collections is the Ideal Test Case
Collections is emotionally charged, heavily regulated, and high stakes in both directions. It’s one of the best test cases we have for AI risk management.
What happens when your AI agent, trying to resolve a tense call, invents a debt forgiveness policy your institution doesn’t have? Or closes an account it decides is a liability?
AI agents are a categorically new kind of system; neither software nor human, something that borrows from both and behaves like neither.
We need to design risk management frameworks that acknowledge this fundamental reality.
