Artificial intelligence in banks now monitors tens of millions of transactions daily, silently in the background. Occasionally, a genuine customer is paying for its error. The question is not whether the machine blundered. Who owns the blame?
What Happens When AI Flags the Wrong Transaction?
The reporting entity needs to ensure the customer is being actively monitored for money laundering and terrorism financing risk.[1] As there’s no way for a compliance team to look at each transaction manually, banks leverage machine learning to score the transaction for risk and detect anomalies.[2]
A small business owner pays an established supplier with the usual money transfer overseas. The amount is higher than usual, the country less familiar, and the model picks it up. The transfer is blocked. This business owner has done nothing wrong and their money is going nowhere until explained.
This is not a theoretical fear. ASIC’s 2024 AI review found a licensee operating a credit risk model that its own personnel conceded was “black box” and that it had no way of identifying the input data that goes into a score or the significance of those inputs to that score.[3] The model was still running months later. Replace credit scoring with transaction monitoring, and the same failure mode appears. A wrong flag is more than an inconvenience. It is a decision nobody can explain.
Who Is Responsible When the System Gets It Wrong?
In that blocked transfer scenario, the AI is not the legal actor. The bank is. No Suspicious Matter Report should be filed until the reporting entity forms reasonable suspicion, that is the responsibility of the reporting entity, not a computer that is producing a risk number.[4] Although the model may focus attention and assign a priority to an alert, whether to escalate, freeze, or dismiss it is an institutional decision.
In my first blog, I made the case that AI in banking only works if the institution can actually account for what the technology is doing and why. A machine supports financial security only if the organisation can explain what was flagged, why, and review that rationale when the customer challenges it. So ASIC’s review concluded that few licensees had developed any process for a customer to appeal against an AI-informed decision, and that for some institutions, it was a challenge even to identify exactly which models they were actually deploying.[5] A system you can’t explain hasn’t just failed you, it’s a system you’ve stopped being able to trust.
What Does Meaningful Human Oversight Actually Require?
ASIC found that human oversight practices by licensees ranged from reviewing each flagged case individually to just a periodic glance at model outputs, and several licensees could not even confirm which approach they were using, meaning there was no consistent standard protecting customers.[6] This discrepancy is important because, where customers are wrongfully flagged, regulations that exist only on paper provide them with no meaningful protection.
Effective monitoring needs certain features. Somebody has to be able to justify why the transaction was flagged. Simply saying the threshold was exceeded explains little.[7] Customers need a process to challenge false positives. A bank must track recurring false-positive trends, because repeated errors may show model defect.
Where I Land?
I am not opposed to AI here. When employed intelligently, it delivers outcomes that humans simply cannot achieve at scale. It also allows human analysts to use their time more effectively by focusing on the cases that genuinely require human judgment.
The trouble was never the transaction monitoring product. It was accountability.
It was in viewing the products as a judgment in and of itself rather than an input to a judgment. That model can prioritize risk and detect an anomaly at speeds greater than the speed of any human being. It cannot, however, assume accountability for that error, challenge its own decision making process or present itself to the regulator. That part will always fall to people. Accountability stays with the institution.
[1]Anti-Money Laundering and Counter-Terrorism Financing Act 2006 (Cth) s 30 (‘AML/CTF Act’).
[2]Caroline I Samson-Onuorah, ‘Data-Driven Fraud Detection Frameworks Integrating Machine Learning, Transaction Monitoring, and Governance Across Modern Banking Platforms’ (2023) 6(2) International Journal of Research in Finance and Management 318, 323.
[3]Australian Securities and Investments Commission, Beware the Gap: Governance Arrangements in the Face of AI Innovation (Report 798, October 2024) 7 (‘ASIC Report 798’).
[4]AML/CTF Act (n 1) s 41.
[5]ASIC Report 798 (n 3) 22.
[6]Ibid 23.
[7]Samson-Onuorah (n 2) 324.
Leave a comment