S2Version 111 models testedUpdated 18 October 2026
Reconciling card and payout income (S2)
Can an AI model reconcile a month of card and payout settlements for a merchant group, find every error, value it and say what to do about it?
Leaderboard
11 models were tested and 9 are ranked. The two Mistral models gave no answer that could be scored. More models will be added to this leaderboard. As of 18 October 2026, Claude Opus 5.5 ranks first, missing $13 a run on average, followed by GPT-5.5 ($137) and Grok 4.7 ($162).
| # | Model | Missed, average | Worst run | Errors found | False alarms | Cost per run | Price in / out |
|---|
Average across Batch (17 errors, $4,832) and Slice (18 errors, $4,853), three runs of each, in US dollars at $1.34 to the pound. The ± figure shows how much a model's result moved from run to run. When two models' ranges overlap, such as GPT-5.5 at $137 ±20 and Grok 4.7 at $162 ±105, the test can't say which of the two is better. Errors found is the average per run, Batch / Slice. False alarms are counted across all six runs. Prices are OpenRouter list prices per million tokens, October 2026.
Background
Reconciliation is how a finance team checks that the money it expected actually arrived. For a merchant taking cards, that means matching what each acquirer settled against what landed in the bank and what was booked in the ledger, with fees, refunds, chargebacks and payouts to third parties in between.
S2 gives each model a month of files for an invented merchant group with a UK company and a Spanish company. The files hold planted errors with known values, plus items that look like errors and aren't, such as settlements due after month end. The model has to read several files at once, match records that use different references, apply the contracts and decide which differences are errors.
Results
Opus 5.5 was the only model to stay under $50 missed in every run. GPT-5.5 and Grok 4.7 came next, then Kimi K3 and Claude Sonnet 4.6. The three cheapest models, DeepSeek V4.1 Flash, GLM 5.3 and Gemini 3.8 Flash, each missed hundreds of dollars on average.
Average against worst run
The worst run shows how far a model can be trusted without checking. Gemini 3.1 Pro, GLM 5.3, DeepSeek and Gemini 3.8 Flash each had a run that missed about $2,600 or more. Grok 4.7 missed the same $58 in all three Slice runs. Switch the chart to Worst run to compare.
The larger Slice files pushed some models back. Kimi K3 went from $151 missed on Batch to $496 on Slice, and DeepSeek from $508 to $1,422. Grok 4.7 went the other way, from $265 to $58.
Errors found least often
Runs that found each error, out of 27 runs by the nine ranked models. The six found least often.
Teal = Batch, navy = Slice. The debit-rate error is in Slice only. The full list of errors stays private, so the test still works when it's rerun.
Economic model
The test measures how much money each model misses. The economic model turns that into money for a particular firm: full cost per reconciliation with review time, written-off errors, late chasing and false alarms included, annual cost, saving against reconciling by hand, payback on the build, and how much cheaper a model would need to be to justify its extra misses. It covers five of the nine ranked models, with an illustrative firm running 200 reconciliations a month.
The price of a cheaper model
Running Gemini 3.8 Flash in place of Claude Opus 5.5 saves 48 cents a run. Each dollar saved costs about $1,000 in missed money and review. None of the four cheaper models in the sample would break even at any price, including free.
The worst run decides how a model is used
Only Opus 5.5 kept its worst run inside a 2% tolerance, so it is the only model that can run without a person reviewing every reconciliation. It comes to about $21 a reconciliation all in, against about $475 by hand.
Cheapest to run, dearest to use
DeepSeek V4.1 Flash costs 4 cents a run and about $230 a reconciliation once misses and review are counted. Gemini 3.8 Flash comes to about $520, more than doing the work by hand.
The Excel model is available on request. In the workbook, write-off share, chase and review time, the by-hand process and build costs are illustrative and move the totals; the order of the models holds across all five scenarios in the workbook. See Method for how the models work.
Example task
One of the planted errors, and what a correct answer looks like. The model gets all the files at once and has to find this error without being told it exists.
- The error
- A €50.35 Bizum payment at the Spanish online store appears in Acquirer B's file of refunds, chargebacks and Bizum and pay-by-bank payments, but has no entry in the group ledger.
- Files to read together
- That Acquirer B file and the ledger for Spain S.L.
- Why it's hard
- The error is an entry that isn't there. The model has to start from Acquirer B's Bizum payments and look for each one in the ledger.
- A correct answer
- A payment with no ledger entry, Spain S.L., EUR 50.35, pointing at the Bizum payment reference, with the action "adjust": book the missing payment in the ledger.
- How often the models found it
- Across all nine models, it was found in 14 of 27 Batch runs and 8 of 27 Slice runs. No error was missed more often.
- What Claude Opus 5.5 returned
- From its first Batch run, word for word. It found this error in all three Batch runs.
{"type": "unrecorded A2A sale", "entity": "ES", "currency": "EUR", "amount": 50.35, "references": ["B-ESO-0806", "L0028346", "P028211"], "explanation": "Bizum sale was settled by the acquirer but is missing from the ledger.", "action": "adjust"}
Method for this test
How every ScalePoint Research test is built, run and marked is on the Method page. These are the details specific to S2.
August 2026 for five sites: a UK restaurant, a UK shop with two concessions, the UK online store, a Barcelona restaurant and the Spanish online store. Two acquirers with different pricing, settlement timing and file formats, plus Bizum, pay by bank and concession payouts.
Batch gives daily batch totals, about 45,000 to 65,000 tokens, with 17 errors worth $4,832. Slice adds every card transaction at the Barcelona restaurant, about 90,000 to 115,000 tokens, with one more error that needs the line detail.
An error counts as found when the answer's references point at it. An amount counts as right within £1 or 1%, whichever is larger. Money missed is the value of every error not found, plus any shortfall on an amount. EUR is put into GBP at 0.85. Money figures are shown in US dollars at $1.34 to the pound. Amounts quoted from the test files stay in pounds or euros.
Each model also scores the share of the money at stake it caught: 100 × (1 − average missed ÷ $4,843). Claude Opus 5.5 scores 100 and Gemini 3.8 Flash 47.
Up to 64,000 output tokens per reply, with thinking capped at 16,000. DeepSeek ignores a token cap, so it ran on medium reasoning effort. A reply with no readable answer counts as everything missed.
The promptShow the exact instructions every model got
Every model got the text below, word for word, followed by the files. The files are not published, because they hold the planted errors.
RECONCILIATION POLICY - THE GROUP (UK Ltd and Spain S.L.)
You are reconciling August 2026 card income and payouts. Compare the acquirer files, payout report,
bank statements, ledger, contracts, chargeback notices and FX reference rates you are given.
Rules:
1. A break is any difference that means money is missing, wrong, unexplained or posted wrongly.
2. Anything due after 31 Aug 2026 under the contract timings (settlements, payouts, chargeback debits,
fee collections) is a timing item, not a break. UK bank holiday: Monday 31 Aug 2026.
Do not list timing items.
3. Acquirer B pays one bank credit per settlement account per value date, which can cover several batches.
Any settlement that arrives late but in full before month end is not a break.
The ledger's daily card takings come from the tills, and its refunds are what the group agreed with
the customer. Where an acquirer figure differs from these, the ledger is right.
4. Differences of up to 0.10 (GBP or EUR) on a single batch are rounding and not breaks.
5. A chargeback with a notice and a debit date after 31 Aug is a timing item.
6. FX: compare conversions with the reference rate for the value date. Inside 0.5% is fine.
Value an FX difference as the loss in GBP against the reference rate.
7. Fees must match the pricing schedule exactly. Value an overcharge as the excess for the month.
8. Actions: "escalate" for anything that needs the acquirer, the bank, a customer or a concession
to act or to be paid, and for any break over GBP 1000 (EUR at the reference rate) even if the
fix is in the group's own books. "adjust" for other corrections the group makes in its own books.
"watch" only for timing items. The limit is checked against each break on its own.
9. Report each problem once, as one break, even if it shows in several files. Report separate problems
separately: one break per batch, invoice, payout or transaction affected. If the same kind of
difference repeats across many transactions, report it once per acquirer account, with the total.
10. Give each break's amount as a positive number in the currency the loss or difference is measured in.
"entity" is the entity whose money or books are affected.
Return only JSON, with this shape:
{"breaks": [{"type": "<short label for the kind of break>", "entity": "UK" | "ES", "currency": "GBP" | "EUR",
"amount": 0.00, "references": ["batch, line, payment, payout, invoice, bank reference or journal ids"],
"explanation": "one short line", "action": "escalate" | "adjust"}]}
Give every id you rely on in "references", so each break can be traced to the files.
List breaks only. Keep explanations short; the answer must fit in one reply.
FILES
===== FILE: acquirer_a_batches.csv =====
[file contents]
===== FILE: acquirer_a_refund_lines.csv =====
[file contents]
===== FILE: acquirer_a_invoice.txt =====
[file contents]
===== FILE: acquirer_b_settlement_summary.csv =====
[file contents]
===== FILE: acquirer_b_report_lines.csv =====
[file contents]
===== FILE: acquirer_b_refund_chargeback_a2a_lines.csv =====
[file contents]
===== FILE: acquirer_b_payout_report.csv =====
[file contents]
===== FILE: bank_uk_ltd_gbp.csv =====
[file contents]
===== FILE: bank_spain_sl_eur.csv =====
[file contents]
===== FILE: group_ledger.csv =====
[file contents]
===== FILE: chargeback_notices.txt =====
[file contents]
===== FILE: contracts.txt =====
[file contents]
===== FILE: fx_reference_eur_gbp.csv =====
[file contents]
This is the Slice prompt. The Batch prompt has the same instructions, word for word, and leaves out acquirer_b_report_lines.csv, the card transaction lines for the Barcelona restaurant.
Rule 8 mentions a “watch” action, but the answer format only allows “escalate” or “adjust”. That is because timing items were not to be listed at all, as rule 2 says.
Differences from the standard method
S2 version 1 ran on OpenRouter's default routes. The standard is zero-retention routes, and results on those routes may differ.
Not ranked
Mistral Large 4 gave no answer in 13 runs on three settings. It kept checking its own working until it ran out of output. It's a public preview released on 6 October 2026, and ScalePoint Research will test it again once the final version is out.
Mistral Large 3 (v25.12) didn't complete because of provider rate limits. Its single check run on Batch found 5 of 17 errors.
Changes
- 25 Oct 2026Excel economic model added: cost per task, payback and scenarios for a given firm.
- 18 Oct 2026Version 1 published with nine ranked models.