S1Version 134 models testedUpdated 18 October 2026
Reading card statements (S1)
Can an AI model read a UK card acquiring statement, find every fee, put each one under the right fee type and flag anything wrong with the statement?
Leaderboard
34 models were tested and 25 gave usable answers. As of 18 October 2026, Grok 4.7 and Claude Opus 5.5 share first place, with almost no fees wrong. Gemini 3.8 Flash is next at 0.37%, about $40 on a typical statement.
| # | Model | Fees wrong | On a typical statement | Price per statement | Weights | Host in this test |
|---|
Fees wrong is the value of fees missed, double counted, misread or put under the wrong type, as a share of the true fees, averaged across 16 statements. A typical statement carries $10,700 of fees. Models within 0.05 points of each other share a rank, and the cheaper one is listed first. Price is the model cost per statement at OpenRouter prices, September and October 2026.
Not usable
Nine models failed the usable bar set before the test: fee lines that add up to the true total on at least 95% of statements, no invented fee lines, and at least three of the four traps caught.
| Model | Fees wrong | Why it failed |
|---|
Background
A card acquiring statement lists what a merchant paid to accept cards that month: interchange, scheme fees, the acquirer's own charges and a long tail of smaller fees. Every acquirer lays its statement out differently, and the same fee can turn up under different names. Reading statements accurately is the first step in any card cost review, and acquirers and brokers read them in bulk to price new merchants.
S1 gives each model the text of synthetic statements from two invented acquirers, one with interchange-plus-plus pricing and one with blended pricing. Every fee has a known right answer.
Results
Grok 4.7 and Claude Opus 5.5 got about $5 of fees wrong across all 16 statements. Five more models stayed under 0.6%. Every usable model found the fee lines and read the amounts correctly, so the gap between models comes from putting fees under the wrong type.
Where the mistakes came from
GPT-5.5 put $1,312 in the wrong place across 16 statements carrying about $171,500 of fees. $906 of that was higher charges on non-UK cards filed as penalties, and $402 was two fees taken out of the settlement recorded as credits.
Price was a poor guide
Mistral Medium 3.5 cost $0.174 a statement and got 1.94% wrong. Mistral Large 3, from the same developer, cost $0.012 and got 0.56% wrong. Claude Opus 4.8 cost more than its successor Opus 5.5 and got 2.16% wrong.
The baseline without AI
A template parser written for both layouts got every fee right on the 12 standard statements, at no cost. It missed two of the four traps. An acquirer layout it hasn't seen would need a new template.
Economic model
The test measures how each model reads statements. The economic model turns those measurements into money for a particular firm: full cost per statement with checking and mistakes included, annual cost, saving against reading statements by hand, and payback on the build. It covers four finalists (Claude Opus 5.5, Grok 4.7, Gemini 3.8 Flash and GLM 5.3), the template parser and a person doing the work by hand, with an illustrative firm reading 10,000 statements a month.
When the parser pays
Sending standard layouts to the template parser saves about $410,000 a year with Gemini 3.8 Flash, which needs spot checks. With Grok 4.7 it costs about $5,000 a year more, because the templates cost more to keep up than the model costs to run.
The prompt changes the order
With the plain prompt, Grok 4.7 is cheapest all in at under 5 cents a statement. With the expert prompt every finalist can run without checks, and GLM 5.3 becomes the cheapest.
Price rises matter less than mistakes
Gemini 3.8 Flash's price is due to double in January 2027. That adds about $3,000 a year for the illustrative firm, because checking and mistakes make up almost all of its cost.
The Excel model is available on request. Its test-based defaults match the calculator in the article; build, upkeep and template costs are illustrative. See Method for how the models work.
Example task
The trap that caught the most models. Each model reads the whole statement and has to return every fee, without being told the trap is there.
- The trap
- On page 3 of a statement for an invented online retailer, the settlement report shows two fees taken out of the money paid to the merchant: £120 of chargeback fees and £30 of next-day settlement fees. They are not in the £3,793.49 of charges invoiced on page 1. The statement prints them as minus figures, because they reduce the payout.
- What the statement says
Net card sales settled 435,397.49 Less: Chargeback fees deducted at source 6 20.00 -120.00 Less: Next-day settlement fee deducted at source 20 1.50 -30.00 Net amount paid to your bank account 435,247.49 Amounts deducted at source are taken from your settlements and are not included in the charges invoiced on page 1.
- Why it's hard
- The two fees look like deductions. A model has to see that money taken from the payout is still a fee the merchant paid, so it belongs in the fees as a positive amount, not as a credit.
- A correct answer
- Two fee lines, £120 for chargebacks and £30 for next-day settlement, both positive and both marked as found in the settlement report, plus an issue saying fees were taken outside the invoice. The merchant paid £3,943.49 in fees, not £3,793.49.
- How often the models got it right
- 8 of the 34 models. The other 26 all found both fees, but recorded them as minus figures, which counts them as refunds of fees rather than fees.
- What Claude Opus 5.5 returned
- The two fee lines, with the other fields left out.
{"description": "Chargeback fees deducted at source", "fee_type": "chargeback", "amount": 120.0, "source": "settlement_report"} {"description": "Next-day settlement fee deducted at source", "fee_type": "fx_settlement", "amount": 30.0, "source": "settlement_report"} - What GPT-5.5 returned
- The same two fee lines, as minus figures.
{"description": "Chargeback fees deducted at source", "fee_type": "chargeback", "amount": -120, "source": "settlement_report"} {"description": "Next-day settlement fee deducted at source", "fee_type": "fx_settlement", "amount": -30, "source": "settlement_report"}
Method for this test
How every ScalePoint Research test is built, run and marked is on the Method page. These are the details specific to S1.
16 synthetic statements from two invented acquirers: 12 standard statements with 30 to 40 fee lines, and 4 with traps such as a fee shown only in a footnote or fees taken out of the settlement. A typical statement is for a merchant taking about $1.3 million a month on cards.
Nine rules, such as record each fee once and keep currencies separate, plus the names of 18 fee types. The same prompt for every model.
Every fee in each answer is matched to the answer key on amount, currency and description. Fees wrong is the value of every mistake as a share of the true fees on the statement, averaged across statements.
Models get the text of the statement, never the page image. Scanned pages are turned into text with Tesseract OCR, so every model gets exactly the same input.
The promptShow the exact instructions every model got
Every model got the text below, word for word, as its instructions, followed by the text of one statement. The statements are not published, because they are the test.
You extract fee lines from a UK card acquiring (merchant services) statement.
Read the whole statement, including summary pages, footnotes, notes and any settlement report. Return every fee charged to the merchant for the statement period as structured data.
Rules:
1. Report the figures printed on the statement. Do not calculate new amounts, and do not adjust any figure to make totals agree.
2. Record each fee once. If the same fee appears in a summary and again in a detail table, use the detail table and do not repeat it.
3. Where one table row carries several fee components (for example interchange and a service charge on the same card type), return one line per component.
4. Credits, rebates and refunds of fees are negative amounts. Show a prior-period correction with prior_period true.
5. Give each line the currency it is charged in. Do not convert currencies.
6. Set source to where you found the fee: main_table, summary, footnote, settlement_report or other.
7. Decode card type codes using any legend on the statement. Leave a card category field null when the statement does not say.
8. Set low_confidence true on any line you could not read cleanly.
9. Add an issue for anything the merchant should know about the statement itself: totals that do not add up, pricing labelled interchange-plus-plus that behaves like a blended rate, fees taken outside the invoice, charges in more than one currency, scheme fees that look marked up, or sections you could not read.
---
Fee types (use exactly one per line):
- interchange: Interchange
- scheme_fee: Scheme fee
- acquirer_service_charge: Acquirer service charge
- blended_msc: Blended MSC
- card_type_uplift: Card-type uplift
- non_qualifying_surcharge: Non-qualifying surcharge
- authorisation: Authorisation
- 3ds_fraud_screening: 3D Secure and fraud screening
- tokenisation: Tokenisation and account updater
- refund: Refund
- chargeback: Chargeback
- terminal_gateway: Terminal or gateway
- pci: PCI
- minimum_monthly_charge: Minimum monthly charge
- account_admin: Account and admin
- fx_settlement: FX and settlement
- rebate_credit: Rebate or credit
- other: Other
Card category fields (use null when the statement does not say):
- scheme: visa, mastercard, amex, other, mixed
- funding: debit, credit, prepaid, mixed
- segment: consumer, commercial, mixed
- region: domestic, uk_eea, international, mixed
- presence: cp, cnp, mixed
Sources: main_table, summary, footnote, settlement_report, other
Issue types: totals_do_not_reconcile, pricing_effectively_blended, fees_outside_invoice, multiple_currencies, scheme_fee_markup_suspected, unreadable_section, other
Return only a JSON object that matches this JSON Schema, with no other text:
{"type": "object", "additionalProperties": false, "required": ["header", "lines", "issues"], "properties": {"header": {"type": "object", "additionalProperties": false, "required": ["provider", "merchant_name", "merchant_ids", "period_start", "period_end", "total_turnover", "turnover_currency", "printed_total_charges"], "properties": {"provider": {"type": "string"}, "merchant_name": {"type": "string"}, "merchant_ids": {"type": "array", "items": {"type": "string"}}, "period_start": {"type": "string"}, "period_end": {"type": "string"}, "total_turnover": {"anyOf": [{"type": "number"}, {"type": "null"}]}, "turnover_currency": {"type": "string"}, "printed_total_charges": {"type": "array", "items": {"type": "object", "additionalProperties": false, "required": ["currency", "amount"], "properties": {"currency": {"type": "string"}, "amount": {"type": "number"}}}}}}, "lines": {"type": "array", "items": {"type": "object", "additionalProperties": false, "required": ["merchant_id", "description", "fee_type", "card_category", "count", "value", "rate_percent", "unit_price", "amount", "currency", "prior_period", "source", "low_confidence"], "properties": {"merchant_id": {"anyOf": [{"type": "string"}, {"type": "null"}]}, "description": {"type": "string"}, "fee_type": {"type": "string", "enum": ["interchange", "scheme_fee", "acquirer_service_charge", "blended_msc", "card_type_uplift", "non_qualifying_surcharge", "authorisation", "3ds_fraud_screening", "tokenisation", "refund", "chargeback", "terminal_gateway", "pci", "minimum_monthly_charge", "account_admin", "fx_settlement", "rebate_credit", "other"]}, "card_category": {"type": "object", "additionalProperties": false, "required": ["scheme", "funding", "segment", "region", "presence"], "properties": {"scheme": {"anyOf": [{"type": "string", "enum": ["visa", "mastercard", "amex", "other", "mixed"]}, {"type": "null"}]}, "funding": {"anyOf": [{"type": "string", "enum": ["debit", "credit", "prepaid", "mixed"]}, {"type": "null"}]}, "segment": {"anyOf": [{"type": "string", "enum": ["consumer", "commercial", "mixed"]}, {"type": "null"}]}, "region": {"anyOf": [{"type": "string", "enum": ["domestic", "uk_eea", "international", "mixed"]}, {"type": "null"}]}, "presence": {"anyOf": [{"type": "string", "enum": ["cp", "cnp", "mixed"]}, {"type": "null"}]}}}, "count": {"anyOf": [{"type": "integer"}, {"type": "null"}]}, "value": {"anyOf": [{"type": "number"}, {"type": "null"}]}, "rate_percent": {"anyOf": [{"type": "number"}, {"type": "null"}]}, "unit_price": {"anyOf": [{"type": "number"}, {"type": "null"}]}, "amount": {"type": "number"}, "currency": {"type": "string"}, "prior_period": {"type": "boolean"}, "source": {"type": "string", "enum": ["main_table", "summary", "footnote", "settlement_report", "other"]}, "low_confidence": {"type": "boolean"}}}}, "issues": {"type": "array", "items": {"type": "object", "additionalProperties": false, "required": ["type", "detail"], "properties": {"type": {"type": "string", "enum": ["totals_do_not_reconcile", "pricing_effectively_blended", "fees_outside_invoice", "multiple_currencies", "scheme_fee_markup_suspected", "unreadable_section", "other"]}, "detail": {"type": "string"}}}}}}
The answer format at the end was also sent as a structured-output setting, for the models that support one. The fee types are names only. The one-line definitions of each fee type were added in the second round, which the article covers.
Differences from the standard method
S1 version 1 ran each model once per statement at temperature zero, on a script written before Inspect was set up. The standard is three runs per model on Inspect. S1 version 1 predates it. Its 0 to 100 score also uses the version 1 rule: 100 for no fees wrong, falling in a straight line to 10 for the usable model with the most fees wrong, and 0 if not usable. Version 2 will score the share of the money at stake a model got right, like S2.
Changes
- 25 Oct 2026Excel economic model added: cost per task, payback and scenarios for a given firm.
- Oct 2026Version 1 published with 34 models, 25 of them usable.