Model tests · October 2026
How ScalePoint Research Decides What to Test
Payments products don't hand a language model a month of files and leave it to get on with the job. Here is which steps the tests give to a model, which they leave to code, and which skills they score.
- In real products, rules and machine learning do most of the work in payments operations. A language model handles the steps they leave.
- Hence each ScalePoint Research test aims at the step a product would actually give to a model, and leaves matching, sums, standard file formats and lookups to code.
- The tests score four of the nine skills most closely: rule application, reasoning across sources, calculation set-up and judgement.
- Each test will also show what a code baseline catches without any model, so readers can see where the model adds something.
A fair challenge to any test of AI models on payments work is that no product runs this way. A reconciliation tool doesn't paste a month of settlement files into a model and ask it to find the breaks. A sanctions screening system doesn't ask a model to compare every customer name with every name on the OFAC, OFSI and EU lists. Rules and machine learning already do that work at almost no cost per transaction.
Rules, machine learning or a model sets out that split across the 51 processes on the ScalePoint Research process map. This piece explains what it means for how the tests are designed.
The design rule
Each test gives the model the step a product would hand to a model, and leaves the steps that code does well to code. A model that matches settlement lines on reference and amount is doing a job a rules engine does for almost nothing, so a test built around that step would measure something nobody needs. The steps worth testing are the ones rules and machine learning leave behind, where a person would otherwise have to step in.
What the tests leave to code
These steps already run on rules, lookups or machine learning in the products ScalePoint Research has looked at, so the tests keep them away from the model wherever they can.
- Exact matching. Pairing a settlement line with a bank receipt on amount, date and reference.
- Plain arithmetic. Summing a file, converting at a given FX rate or applying a known fee formula.
- Standard formats. Reading CSV exports, MT940 or camt.053 bank statements and acquirer files with a known layout.
- Lookups. Interchange tables, BIN ranges and scheme time limits for chargebacks.
- High-volume scoring. Fraud scores, transaction categories and merchant category codes, where a trained model gives an answer in milliseconds.
- Name matching in screening. Comparing names against sanctions lists, including names spelt differently, which is the screening engine's job.
What the tests give to the model
Nine skills for testing AI models in payments lists the skills a payments task can ask for, from extraction to staying within authority. The tests score four of them most closely, because they make up the work rules and machine learning leave behind.
| Skill | What the model has to do | Example |
|---|---|---|
| 3. Rule application | Work out which contract term, scheme rule or regulation applies to this case, and what it means | Whether a payment falls under an SCA exemption under PSD2 or the UK rules |
| 4. Reasoning across sources | Connect evidence that no single file holds | A shortfall that only shows once the acquirer file, the bank line and the ledger are read together |
| 5. Calculation set-up | Choose the right figures and the right formula before any sums are done | Which rate in the contract applies to a commercial card from outside the UK |
| 6. Judgement | Decide what matters and what to do about it | Close or escalate a sanctions alert, with a reason a reviewer can check |
Extraction and classification still appear in the tests, because a model has to read a document before it can do anything with it. The tests tag and count them as well. Long context comes from the size of the files in a task, and tool use and staying within authority come in with the agent tests.
What judgement covers
Judgement is the broadest of the nine skills, so the tests break it into five kinds of call.
- Spotting what doesn't add up. Noticing a problem nobody asked about, such as a subtotal that doesn't match its lines or a fee netted out of a settlement without being shown.
- Weighing conflicting evidence. Deciding which source to trust when two disagree, such as an acquirer report against the bank statement, or a customer's account against the transaction record.
- Judging materiality. Telling a real loss from a timing difference that will clear by itself, and putting the largest problems first.
- Knowing when not to decide. Referring a case to a person when the evidence is thin. Closing a sanctions alert wrongly costs a firm far more than sending it to an analyst, so the tests count referrals separately from errors.
- Not being misled. Holding firm against a document or claim built to mislead, such as an altered invoice at onboarding or a chargeback filed with a false reason code.
A code baseline next to each test
Each test page will also show what plain code catches on the same files with no model involved. For a reconciliation test, that means a rules-based matcher. For a statement test, a parser built for the known layouts. The baseline shows readers how much of the task rules already handle, and the model results then show what a model adds on the steps that are left.
The baseline is also a check on each test. If code catches nearly everything, the task needs more judgement in it before it says much about the models.
Last reviewed 21 October 2026.
Part of ScalePoint Research. Related reading: