How the testing works

• Test data. The data is synthetic, built around one invented market: a restaurant and retail group, the two acquirers it uses and, in later tests, their other merchants. No real card or customer data is involved, and the same market is reused across tests. Every case has a right answer set before any model runs. A sample of cases is published with each paper and the rest stays private, so it can't end up in a model's training data.
• Runs. Every model gets the same plain prompt and the same fuller prompt with definitions and rules. Tasks with several steps are run three to five times per model.
• Models, not apps. Each test calls the models directly through their APIs, with the same instructions and tools for every model. Apps and add-ins built on these models, such as Claude in Excel, add their own instructions and features, so a model can do better or worse inside them than in my tests.
• Results. Each result shows accuracy, model cost per task and the cost of mistakes, with the model version, run date and provider.
• Data rules. Each result also shows whether the model can be hosted in the EU, whether zero data retention is offered and whether the weights are open.
• Funding. No model maker pays for a test or sees the private cases.

Changes to this method will be listed here with their dates.