Banks are guarding against the AI they control. The real threat is the AI they can’t. Philip Dutton explains the dangers of Shadow AI to the world of finance.
When Andrew Bailey, Governor of the Bank of England, recently told the UK’s Treasury Select Committee that AI in banking was in danger of going ‘off-piste’, he raised questions around hallucination, misconduct and legal liability. But this only accounts for the AI they can see.
The banking industry has been one of the fastest and most ambitious adopters of AI, and it will see massive leaps in efficiency and cost-effective financial advice as a result. Only last month, Sheldon Mills, of the Financial Conduct Authority, spoke with the Financial Times about how the use of AI could lead to a revolutionary democratisation of financial advice and, through new cost efficiencies, a better banking experience for everyone.
This is true, but in order to realise this AI-enhanced financial world, the industry first needs to get its house in order when it comes to which tools are being used, and keeping track of what they are being used for.
The governor’s advice on this was pragmatic: “you’ve got to be very clear where the legal liability rests in all of this.” I couldn’t agree more. And from an auditing point of view, the decades of adjustments that banks and financial institutions have made to adapt to new, post-crash governance and regulation, like BCBS-239 or DORA, are now at risk of being undermined if the data that has led to the decisions is traceless in the first place.
However, to take Governor Bailey’s assessment even further – what if the AI tools themselves are traceless? How can you even begin to audit decisions?
What Survivorship Bias Teaches Banking
In World War II, the statistician Abraham Wald made a famous discovery. The British military wanted to improve the armour on their aircraft, so they studied where returning planes had been hit most and reinforced those spots. Wald pointed out the flaw: they were only looking at the planes that made it back. The ones that mattered, the planes hit in the places that brought them down, weren’t in the hangar to be studied. He called this phenomenon ‘survivorship bias’.
I see AI in banking heading the same way. It is all well and good to study the fleet that comes back, the approved agents and internal AI platforms that we control, but the real threat is the AI nobody is logging, and nobody can see.
What I am referring to is ‘shadow AI’: the use of AI tools that an organisation has no control over. It was not too long ago that the buzzword for CISOs was shadow IT, its unsanctioned-software counterpart. Training was put in place, guardrails were established, and for the most part, that felt like problem solved.
The issue is that where shadow IT was largely about controlling which tools and systems employees were using, shadow AI changes how decisions are being shaped. These actions aren’t just untraceable because they sit outside your field of view; the problem is compounded because there is no way of knowing what is happening to your data while it is in there.
Depending on the tool and its terms, data may be processed across multiple data centres or jurisdictions, and in some cases retained or used to improve larger systems. Worst of all, what returns to your firm’s systems may, to the naked eye, appear plausibly linked to the original input, but from a lineage perspective, it represents a distinct, untraceable and therefore untrustworthy discrepancy.
Losing the Data Trail
Auditors are increasingly knocking at firms’ doors, seeking tangible evidence of how decisions have been made. Traditionally, firms have provided evidence through data lineage tools that map the journey of data through your business. You can see where the data originated, and ultimately, how it led to the decision on the other end.
The problem is that, increasingly, staff who are under pressure to produce results quickly, and often encouraged to use AI — with little to no training on what that should look like — are using external AI tools to get the job done.
As Governor Bailey suggested, this is not so much of a problem when the AI is being used to draft a quick email or do some fact-checking. But the industry seems intent on growing the use of AI within its walls, and at some pace. Once tools begin to ingrain themselves more deeply in the high-stakes decision-making that a bank or trading floor might be doing, you now have autonomous decisions being made with no way to explain them to regulators or to the people they affect. Feed those decisions on flawed or unrepresentative data and you’ve essentially automated financial discrimination at scale.
This isn’t to say we should limit our horizons. AI can deliver on all that it promises, but like in a high-end restaurant kitchen, if the ingredients and environment set up around these ambitions are not of the highest quality, the end product will always disappoint.
Why Bans Don’t Work
You might be tempted to say, “well, why doesn’t the banking industry just do more to block this?” The truth, however, is that these tools are here to stay, and for good reason. If you can’t provide a safe, workable in-house alternative, your employees will find themselves having to turn to tools outside of your organisation in order to bridge that gap.
People don’t necessarily want to use shadow AI, and blocking and training can be of some use, but often staff find they have to use external tools, either because they don’t have access to a safe alternative or because the tool they do have is too underpowered for the task at hand.
This means you lose a level of oversight and control, yes, but it also means your business is running on systems that are shop-bought, not tailored. Where there are shadow AI tools, there are unoptimised processes, and limited possibilities for where AI integration can take you.
This is a problem for all key stakeholders. Firms in the industry need to do more to educate their staff on safe usage of AI, and upgrade their internal offerings.
The other problem that comes with bans is the unwanted effect of pushing shadow AI into shadow IT as well, as staff begin to bypass blocks by using their own personal devices. The result is a babushka doll of systems and data processing that exists entirely out of your control or visibility. Pushing this activity underground, and into a culture of open secrets and underreported use, will only worsen the problem and slow any potential solution.
How We Move Forward
So, Pandora’s box is open; what now? For the time being, Governor Bailey’s pragmatic advice should be heeded. We must do more to assign legal responsibility to individuals. If I cannot tell how a decision was made through its data’s lineage, then we can at least do more to point to who set those decisions in motion. The so-called ‘human-in-the-loop’ is by no means a perfect ethical solution, and the AI industry as a whole will need to do more when it comes to assigning a working framework for who is responsible for the automated aspects of its functions, but, for now, it can help steady the ship.
Over the next few years, the shadow AI threat will only grow in scale and sophistication, as the big players in the consumer market continue to advance.
Regulation will follow, as it always does. As governance matures, we should see more structured processes emerge for dealing with shadow AI, but tighter rules alone won’t be enough. They must be matched by serious investment in embedded, traceable tools. And firms should resist the temptation to lean on any single governance framework. The stronger approach is to combine risk-based guidance with legal requirements, then build those rules directly into internal controls for model risk and third-party oversight.
For security and compliance leaders, the key will be to avoid blanket bans or witch hunts, and instead take a more realistic approach: allowing external AI for low-risk tasks, while drawing a hard line around exporting sensitive data or using AI where privacy is paramount.
The banking industry will act as a kind of proving ground; the lessons it learns now will land on every regulated, data-driven industry soon enough. It is a hard balancing act, one that demands a constant eye on shifting regulation while still innovating at a sensible pace. Get it wrong in either direction, through too much freedom or too much control, and shadow AI wins either way.
