← Writing

Buying Payments and Finance Operations Software With AI Inside: What the Contract Doesn't Check

Vendors increasingly claim AI or agent features. The usual buying process says little about how good that AI is, and a test on the buyer's own cases can.

In brief

When a bank, a fintech or a large merchant buys software for its payments or finance operations, the vendors pitching it are increasingly claiming AI or agent features.

Procurement will typically still run the purchase like any other software buy, with an RFP, a vendor questionnaire, a security review, references and a contract with an SLA. But how far do those steps test how good the vendor's AI actually is?

That question matters more each year, since the AI and agent parts increasingly decide whether the product gets the answer right: whether it catches the reconciliation break, files the right chargeback or clears the right sanctions alert. This article looks at what buyers check today and argues that the missing piece is a test of the AI inside the product, run on the buyer's own cases. The examples come from the ScalePoint Research process map, which rates 51 payments and finance processes for how well AI agents fit them.

Buyers know the claims are hard to check

In July 2026 The Hoffman Agency published a survey of 450 senior technology buyers in the UK, US, France and Germany. The respondents buy technology for their whole business, and the findings fit operations software well. 64% said AI purchases get more scrutiny than other technology, 78% said many AI vendors make claims that are hard to verify or compare, and 61% said they're unsure which AI products are credible.

When claims are hard to verify, 46% said the decision takes longer, 37% said they favour larger, established suppliers and 32% said they add procurement or compliance checks. 27% said purchases get delayed and 16% said deals get abandoned. None of those responses tells the buyer whether the AI in the product will work on its own cases.

Asked what increases their confidence in a supplier, buyers pointed to transparent explanations of how the product works (46%), customer case studies with measurable outcomes (41%) and proof of deployment at scale (40%). All three come from the vendor or its existing customers.

What the buying process checks

One industry guide puts a typical AI vendor assessment at 40 to 80 questions. In the published guides and templates ScalePoint Research has looked at, the questions mostly cover governance, data use and security, and don't appear to probe in much detail how well the AI performs.

The contract rarely fills the gap. Vendors often disclaim responsibility for the accuracy of outputs and offer SLAs that cover uptime only. A Stanford study found that 17% of AI vendor contracts warranted the product would perform as documented, against 42% of ordinary SaaS contracts. Vendors argue that a model's answers vary from run to run, so a fixed error rate is hard to promise. Hence the contract often has no accuracy promise at all.

For EU banks, payment institutions and e-money institutions, DORA already asks for more. Since January 2025, a contract with a technology supplier that supports a critical or important function has to include precise quantitative and qualitative performance targets, and give the firm the right to monitor the supplier's performance on an ongoing basis. For AI-enabled operations software, accuracy is the obvious target to include. Merchants aren't covered by DORA, but nothing stops them asking for the same.

The EU AI Act matters less here than people tend to expect. In financial services its high-risk rules cover credit scoring of individuals and life and health insurance pricing, and expressly leave out fraud detection, so most operations tools fall outside them. Those rules now start on 2 December 2027, after the Digital Omnibus delayed them. What applies today is a duty to support AI literacy among staff and, since August 2026, to tell people when they're dealing with an AI system.

How much the results can vary

In payments and finance operations, a wrong answer costs money. If a reconciliation misses a break, the money usually isn't recovered. If a chargeback response is filed badly, the dispute is lost. If a sanctions alert is cleared that shouldn't have been, the firm has a regulatory problem, and if invoice processing misses a duplicate supplier invoice, the money goes out twice.

ScalePoint Research has been testing AI models on a card reconciliation task: a month of settlement files, bank lines, a ledger and a contract for an invented merchant, with planted errors and known answers. Every model got the same files and the same instructions. The best one missed about £10 of errors a run. Others missed several hundred pounds a run on average, and some single runs missed more than £2,000. The same model could also find every error on one run and miss a large one on the next.

Two vendors could both say they offer AI-powered reconciliation and sit at opposite ends of that range. A vendor can also move to a different model after you sign, to cut its own costs or because the old one is retired, and the AI's accuracy moves with it. A questionnaire answered once at the start won't show either of those.

Testing closes the gap

The buyers in the Hoffman survey said what would help: 37% cited demonstrated technical performance benchmarks, and 36% said independent third-party validation helps build trust. For operations software, the benchmark that counts is how the AI inside the product performs on the buyer's own work. That means testing the AI part on its own: the model or agent that reads the files, matches the records or decides the case. Procurement already checks the rest of the software. A useful test has four parts:

The test also gives the contract something to hold on to. Lawyers now advise buyers to ask for service levels tied to outcomes, such as 99% of invoices processed correctly, with a right to check performance against them. One compromise ties the warranty to benchmarks set out in an exhibit and measured over a large enough sample of cases. A test set with known answers is what makes a clause like that, or a DORA performance target, possible to check.

Questions to ask before you sign

  1. What measure of accuracy will the vendor commit to, and is it in money or in cases?
  2. Has the AI been tested on your own cases, or a realistic set built from them, with answers you already know, or only on the vendor's demo data?
  3. How much do results vary between runs on the same case?
  4. Which model does the AI run on, and does the vendor have to tell you when it changes?
  5. Will the vendor put the measure and the test set in an exhibit, with a right for you to rerun the test after updates?
  6. Who on your side owns the test and the results once the contract is signed?

Sources

Part of ScalePoint Research, which publishes its own tests of AI models on payments tasks:

martin@scalepointpartners.com