We don't fully know what frontier models can do

I find it fascinating that we don't fully know what frontier models can do. I kept coming back to this while building my first benchmark test for a payments task, 34 models reading UK card acquiring statements.

The classic example is from 2022. Researchers found that adding “Let's think step by step” to maths questions took a model's accuracy on a standard benchmark from 18% to 79%. The model was unchanged. One extra line of prompt brought out a reasoning ability nobody knew was there.

The one I find most intriguing is from 2025. Anthropic looked inside Claude while it was asked to add two numbers. When asked, Claude said it used the carry method taught at school, but its internals showed something quite different. One path worked out roughly how big the answer would be, another worked out just the last digit, and the two were combined. So even the model couldn't explain how it did it.

Nobody designs these abilities one by one. They emerge during training and are fixed when the model is released, and the researchers then test the finished model to find out what it can do. A benchmark only covers what someone thought to test for, and as models get more capable, the range of things they might do grows faster than anyone can test.

Hence I think a good share of what today's models can do is still undiscovered, and some of it will be found by people testing them on real work. My own test turned up a few surprises along those lines, which are in the S1 write-up.