Research / Model tests
Model tests
Each test runs the same payments task through every model and scores what its mistakes cost in pounds against what it costs to run.
Models tested34
Finalists7
Fewest fees wrong0.03%four finalists, expert prompt
Cheapest finalist$0.005per statement, DeepSeek V4.1 Flash
Result
Weights
Developer based in
Sort by
Showing 34 of 34 models.
Fees wrong is with the plain prompt. The expert prompt was run on the seven finalists only.
| # | Model | Result | Fees wrong | Cost per statement | Expert prompt | Weights · based in |
|---|
Being built
Reconciling card and payout income
Each model gets a month of an invented merchant group's settlement files, bank lines, ledger and acquirer contract, and has to find and value every difference. The data has known breaks planted in it, plus timing differences that shouldn't be flagged. Models are scored on money missed and false alarms, against cost per run, over three runs.
The leaderboard for this test appears here once it has run. Read about the reconciliation cluster →