What the test asks
An AI agent is a model that works through a task in steps, deciding for itself which tool to use next. In this test the agent fetches the files, matches sales to settlements and settlements to bank credits, finds the differences, puts a value on each one and decides which to escalate to a person. That takes somewhere between 30 and 100 steps, so a mistake early on carries through to the result.
The question is the same as in Paper 1: which model is good enough for the job, and what does it cost? Paper 1 found that with good instructions, a model at half a cent a statement matched one at 25 cents. I expect bigger gaps between models once the work takes many steps, and this test measures those gaps in money.
Why reconciliation
- Every difference in the data has an exact right answer, so the scoring needs no judgement.
- The result comes out in money: what each missed difference would cost the business.
- Finance teams at merchants and payments firms reconcile card income every month, across files from acquirers, the bank and their own systems.
The test company
The test runs on an invented UK restaurant and retail group taking about £1 million a month on cards through two invented acquirers, the same ones used in Paper 1. The company has a settlement file for every batch, monthly acquirer statements, contract price schedules, a bank account, a sales ledger and an email inbox. All of it is synthetic and marked as test data. The company is built once and reused for later Labs tests.
What the agent can reach
The agent works in a sandbox and reaches the company's systems only through tools: the file store, the bank, the ledger, the inbox, a Python workspace for matching thousands of rows, and a reconciliation workbook for its results. It can also draft an escalation, request approval or post an adjustment. It gets a one-page reconciliation policy that sets the matching tolerances, a materiality threshold and the rule that larger adjustments need approval first.
What's hidden in the data
Each scenario carries planted differences with known answers, in four groups.
- Money that went missing or went out twice, such as a settlement that never reaches the bank or a refund paid twice.
- Charges the contract doesn't allow, such as one card type charged above the agreed rate all month. Finding it means reading the contract and the statement together, which is the skill Paper 1 tested.
- Items that look like problems and aren't, such as month-end sales settled the following month, or a short settlement explained by an email in the inbox. Escalating these costs a person's time.
- Requests the agent should refuse, such as an email asking for the settlement account to be changed.
The full list stays private, so that no model can be tuned to the test.
How it's scored
- Money missed: the value of every difference the agent missed or got wrong. This is the headline measure, in the same role as fees wrong in Paper 1.
- Differences found out of the total, so one large miss doesn't hide how the agent did on the rest.
- False alarms: each item escalated that didn't need it, valued at 30 minutes of an analyst's time.
- Unsafe actions: posting an adjustment above the threshold without approval, booking unexplained money as income, or acting on the suspicious email. Any one of these fails the run.
- Reliability: the share of scenarios where the agent got everything right on three runs out of three. Agents vary from run to run, so one good run says little.
- Cost and time per completed reconciliation, from actual usage.
The marking is done from the workbook against the answer key, with no AI model doing the marking. A rules-based matching script, which is how reconciliation software matches today, is scored the same way as a baseline.
Models and instructions
The test covers the seven Paper 1 finalists (Claude Opus 5.5, Grok 4.7, Gemini 3.8 Flash, Mistral Large 3, GLM 5.3, GPT-5.5 and DeepSeek V4.1 Flash), plus a few newer models chosen when the test runs. Each runs under zero data retention terms, with the exact model version recorded.
As in Paper 1, the models get two levels of instructions: the task and the policy alone, and the same plus an expert briefing on reconciling card income. Hence the test also shows whether good instructions still close the gap between cheap and expensive models when the work takes many steps.
After this test
The same company runs the next two tests. In one, an agent decides which chargebacks to fight and gathers the evidence. In the other, an agent reviews six months of card statements against the contract. The same setup also runs on a business's own files as a private evaluation.
If your team reconciles card income and you'd like to see how a model does on your own files, or you'd like to suggest a task for a later test, let me know at martin@scalepointpartners.com.