Research / Where models fail

Where AI models go wrong in payments work

Every test is broken into checks: one fee line, one planted error, one trap. Each check is tagged with the skill it needs and, where it needs payments knowledge, the area of payments it comes from. This page adds up the misses across every test so far, for three leading models (Claude Opus 5.5, GPT-5.5 and Grok 4.7) and for everyone else.

By skill

Share of checks missed, by the main skill each check needs. Each test is shown on its own, because the tests differ a lot in size.

By area of payments knowledge

Only checks that need payments knowledge. Fee and card classification is mostly the payments vocabulary on a statement, such as uplifts and non-qualifying surcharges. Settlement and netting covers fees taken out of a settlement, settlement timing and money paid to the wrong account.

Payments-specific or general

By size of the error

Value of the fee line or planted error, in US dollars at $1.34 to the pound.

What stands out so far

• The top models read well and classify well. Across 1,770 fee lines they misread none. The only classification misses by Claude Opus 5.5 and Grok 4.7 in Test 1 were two small late settlement surcharges, filed as FX or other. Whether a late settlement surcharge counts as a non-qualifying surcharge is a UK acquiring convention. GPT-5.5 also filed non-UK card surcharges as non-qualifying surcharges, the most common mistake across all models.
• Payments-specific checks were missed more often than general ones in both tests, about twice as often in Test 2, for those three models and for everyone else.
• Settlement and netting, and local payment methods, are the weak spots. Fees taken out of a settlement and a Bizum payment with no ledger entry were missed far more often than anything on the face of a statement.
• Those three models never missed a check worth over $1,000. Their misses were small items that need a contract term or two files read together.

How the tally works

A check is one thing a model can get right or wrong in one run. In Test 1 each fee line counts twice, once for reading it and once for its fee type, and each trap and faulty statement counts once. In Test 2 each planted error counts once per run. The three leading models are Claude Opus 5.5, GPT-5.5 and Grok 4.7. They were picked from the overall Rankings in October 2026 and stay the same across tests until the group is reviewed after Test 3. Other models are the remaining usable models in Test 1 and ranked models in Test 2. The tags come from ScalePoint Research and the counts are generated from the run files. With two tests, some cells rest on a handful of checks, so every cell shows its count. The detail for each test is in its article.