The aim is to show which model to use for which task, weighing performance against price, and to test how well agent workflows hold up in finance work. In an agent workflow, a model works through several steps on its own.
Who it's for
- Finance, operations and revenue leaders choosing which model to use for a task. Prices vary widely between models that can do the same job. In the first test, a model at half a cent per statement matched one at 25 cents once both had good instructions.
- Risk and compliance teams who have to know where a model runs and what happens to the data before it's used.
General benchmarks test maths, coding and general knowledge. They don't test real payments work, so a firm can't tell from them how reliable a model would be on sensitive tasks in payments and finance operations.
Two ways to use it
- Public benchmark. Results from each test are published on this site and are free to read. See the latest results.
- Private evaluation. An advisory engagement that runs the same method on a firm's own documents and workflows. The results stay with the firm. More on private evaluations.
How the testing works
- Test data. The data is synthetic, built around an invented UK restaurant and retail group that's reused across tests. Every case has a right answer set before any model runs. A sample of cases is published with each paper and the rest stays private, so it can't end up in a model's training data.
- Runs. Every model gets the same plain prompt and the same fuller prompt with definitions and rules. Tasks with several steps are run three to five times per model.
- Results. Each result shows accuracy, model cost per task and the cost of mistakes, with the model version, run date and provider.
- Data rules. Each result also shows whether the model can be hosted in the EU, whether zero data retention is offered and whether the weights are open.
- Funding. No model maker pays for a test or sees the private cases.
Changes to this method will be listed here with their dates.
If you'd like to suggest a task for a later test, or see how a model does on your own files, let me know at martin@scalepointpartners.com.