Nine Skills for Testing AI Models in Payments and Finance Operations
Test 1 asked for the two easiest skills, so with good instructions the models came out level. Here is the full list, and where each test fits.
- A test only separates AI models if the task asks for something some of them can't do.
- Test 1 asked for extraction and classification. With the expert prompt, all seven finalists got almost every fee right.
- The one part of Test 1 that needed judgement, spotting the two faulty statements, is where the best models pulled ahead.
- Test 2 adds reasoning across sources and long context. Later tests add rule application, calculation and judgement.
In Test 1, 34 AI models read UK card acquiring statements and had to list every fee with its amount and type. With a plain prompt, the results spread out. With an expert prompt of about 1,500 words, the seven finalists finished within one or two fees of each other across 22 statements, and the cheapest cost half a cent a statement against 25 cents for the most expensive.
That's a useful result for anyone reading statements at volume. It doesn't say much about which model to trust with harder work, because the task didn't ask for much. Reading a statement is mostly finding the lines and putting each one in the right box.
The nine skills
Payments and finance work asks a model for a mix of skills. I use nine to plan the tests, from the most mechanical to the ones that need the most judgement.
| Skill | What it means in payments work | Tested in |
|---|---|---|
| 1. Extraction | Finding the data in a document, such as the fee lines and amounts on a statement | Test 1, Test 2 |
| 2. Classification | Putting each item under the right type, such as a non-UK card surcharge rather than a penalty | Test 1 |
| 3. Rule application | Applying a contract term or scheme rule, such as checking a fee against the contracted rate | Later tests |
| 4. Reasoning across sources | Tying several documents together, such as a settlement file, the bank lines and the ledger | Test 2 |
| 5. Calculation set-up | Building the right calculation, such as the blended rate on a pricing deal | Later tests |
| 6. Judgement | Deciding what matters, such as which difference is a real error and which will clear by itself | Later tests |
| 7. Long context | Working across a lot of material at once, such as a month of files | Test 2 |
| 8. Tool use and planning | Looking things up step by step in other systems | Not planned for now |
| 9. Staying within authority | Not changing what it isn’t allowed to change | Not planned for now |
The last two matter most for AI agents, which work a case step by step in a firm's own systems. ScalePoint Research has dropped agent tests for now, so those two wait.
What Test 1 showed about the skills
Every usable model found the fee lines and read the amounts correctly, so extraction wasn't the problem. The mistakes were classification, mostly higher charges on non-UK and commercial cards filed as penalties. A payments analyst learns those calls on the job, so they went into the expert prompt, and the gap closed.
Two of the statements were faulty: one had a subtotal that didn't add up. Spotting that is a small piece of judgement, because the model has to check the statement's own sums rather than copy them out. Claude Opus 5.5 and Grok 4.7 flagged both faulty statements at every prompt level. The cheaper finalists only flagged them with the expert prompt, and GPT-5.5 still missed the faulty subtotal.
What changes in Test 2
Test 2 gives each model a month of an invented merchant group's settlement files, bank lines, ledger and acquirer contract. The model has to find and value every difference. That adds reasoning across sources and long context. It also asks the model to leave timing differences alone, such as a card sale on 31 October that the acquirer settles on 2 November, which is a first step towards judgement.
Later tests move further down the list, to applying contract terms, setting up a pricing calculation and deciding which alerts are real.
What this means when choosing a model
A model that does well on a reading task has shown it can read. It hasn't shown it can apply a rule or make a call. Hence a business should test each model on the skills its own task needs, and a price gap measured on an easy task shouldn't be assumed to hold on a harder one.