A model that's good value on one task can fail on the next

I tested AI models on reading card statements, and Google Gemini 3.8 Flash was the best-value model, close to the top on accuracy at a fraction of the price. Then in my second test, on reconciling a month of card and payout settlements, it missed $2,569 of $4,843 in planted errors per run. Anthropic's Claude Opus 5.5 missed $13.

So a model that's good value on one payments task can't be assumed to hold up on a harder one, and the newest version isn't always the right pick. Gemini 3.8 Flash only came out in September and is a newer release than Gemini 3.1 Pro. It's a smaller and cheaper tier, though, so it's perhaps not surprising that it did worse on a more complex task.

I'm running a series of 11 tests under ScalePoint Research on how AI models handle payments and finance operations work. I mapped 50 payments and finance processes into 11 clusters, such as reconciliation, disputes and onboarding, and each cluster gets one test. Whether a firm buys a product or builds its own, LLMs will be part of a hybrid solution where code does the matching, sums and lookups, and machine learning does the scoring. Hence each test focuses only on the steps an LLM would actually be used for, and on the skills those steps need most: reasoning and judgement, such as applying a contract, reasoning across several files and deciding what matters.

Frontier models are doing best so far. I'm most interested in where even the best still miss. In the second test, checks that need payments knowledge were missed about twice as often as general ones, and settlement netting and local payment methods such as Bizum also came out as weak spots.

Chargebacks are next on the test roadmap. After the 11 cluster tests, I plan to move from single-step to multi-step agentic workflows. If you have a payments operations, risk or finance task you'd like to see tested, happy to hear about it. Let me know.