Payments, finance and risk benchmarks for AI labs and agent builders

Written by practitioners from real payments, finance and risk operations work, kept private, and scored in money, so you see what every mistake your model makes would cost.

Why these operations need their own tests

In payments, finance and risk operations, every wrong answer has a price

A missed refund, a chargeback fought on the wrong evidence, a sanctions alert cleared without proper checks or a fee read off the wrong line costs a firm real money. Hence every case we write comes with an answer key and a scoring rule that values each mistake, and we report the money a model loses alongside how often it gets the answer right.

Sample case

One planted error from the settlements test

This is one of 17 errors hidden in a month of card and payout files for a UK and Spanish merchant group. The model gets every file at once and isn't told the error exists.

Reconciling card and payout income (S2)Reconciliation · synthetic data · single reply, no tools

The error

A €50.35 Bizum payment at the Spanish online store appears in Acquirer B's file but has no entry in the group ledger.

Why it's hard

It looks like a basic check, but the error is an entry that isn't there. The €50.35 payment sits in a file that mixes refunds, chargebacks, Bizum and pay-by-bank payments, so the model has to take each Bizum payment and look for it in the ledger. Comparing the entries it can see won't find the gap.

The correct answer

A payment with no ledger entry, Spain S.L., EUR 50.35, pointing at the Bizum payment reference, with the action to book the missing payment.

How often models found it

14/27runs, daily batch files
8/27runs, with every transaction

This was the error the models missed most often in the whole test.

{"type": "unrecorded A2A sale", "entity": "ES", "currency": "EUR",
 "amount": 50.35, "references": ["B-ESO-0806", "L0028346", "P028211"],
 "explanation": "Bizum sale was settled by the acquirer but is missing from the ledger.",
 "action": "adjust"}
Claude Opus 5.5's answer, word for word. It found this error in all three runs. See the full test →

Results so far

What the published tests show

What we offer

Four ways to work with us

01

Private benchmark runs

Run your model or agent, including unreleased versions, on our held-out cases. You get the score, the money lost per 1,000 cases, the cost per case and every case it got wrong. Results stay with you.

02

Expert-written tasks

New cases and scoring rules written to your spec, for your own evals or training environments, covering processes such as chargebacks, reconciliation, onboarding and sanctions alert review.

03

Independent evaluation

A test of what your product claims on accuracy or cost, with the method agreed with you. Anything published is labelled as commissioned and includes the method.

04

A test suite on your clients' cases

For agent builders: a suite built on anonymised cases from a client deployment, so you can rerun it whenever you change model and show the client that accuracy has held.

The pilot

A free pilot on twenty cases, with results in two weeks

The pilot is free. Pick a process, and we run your model on 20 private cases from it. You see the item-level results before deciding on anything bigger.

Day 1

Scoping call

Agree the process, the model and how you want results reported.

Day 3

Cases chosen

Twenty cases picked from the private set, with the scoring rule shared.

Day 8

Runs and marking

Your model runs on a zero-retention route and every answer is marked.

Day 14

Results

Scores, money lost, cost per case and each case it got wrong.

How the cases are built

Written by people who have run these operations

Questions

Common questions

Can we use you alongside our existing eval provider?

Yes. We cover UK and European payments, finance and risk operations in depth, so most firms would use us for that part of their testing.

Do vendors pay to appear in your public tests?

No. ScalePoint Research picks the models in its public tests. Vendors never pay to be in them or see results early, and paid work has no influence on them. Commissioned work is labelled as commissioned and includes the method. The full rules are on the method page.

Which processes do you cover?

The process map covers more than 50 payments and finance processes in 11 clusters, from reconciliation and disputes to onboarding and sanctions screening. Published tests so far cover card statements, reconciliation and judgement calls across nine processes.

How is our data handled?

Models run on zero-retention routes as standard. Results from private runs stay with you and aren't published.

Test your model on payments, finance and risk work

Tell us which process you want to test and we'll set up a free pilot.