A while ago I wrote that LLMs are too positive, and that if you want a model to judge your work you should ask it for the veto rather than the approval. I still think that. But a paper I read this month made me realise the judge has a second failure mode, and it points in the opposite direction.
The paper is “Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification” by Fateme Mazdarani and Carlos Toxtli. The setup is deliberately simple. Take linear equations. Ask a model to do three things: say whether the final answer is right, say whether each step is right, and point at the first wrong step if there is one. Then feed it two kinds of solutions. The canonical one, the way a textbook would write it. And perturbed ones, which reach the same answer by a different but equally valid route.
The models did fine on the canonical traces. On the valid perturbed ones, the base models rejected between three quarters and five sixths of them. Correct solutions, marked wrong, because they didn’t look the way the grader expected.
The grader has taste, and taste is not correctness
That is the part I keep coming back to. The grader was not checking logic. It was checking resemblance. It had a picture of what a solution looks like, and anything off that picture got a fail.
If you have ever had code review from someone who marks things down for not matching their style, you know the feeling. The difference is that a human reviewer usually knows they are doing it. The model does not. It returns a false with the same confidence it returns a true.
The authors’ conclusion is the one I want to underline: evaluator robustness has to be measured separately from solver accuracy. A model that solves the problem well is not automatically a model that grades the problem well. They are different jobs, and the second one has a bias the first one hides.
Why this matters outside of algebra
I don’t grade linear equations. But I do run pipelines where one model checks another. Code review agents. A second pass that verifies a first pass. Multi-agent setups where a planner’s work is approved by a critic before anything ships.
Every one of those is a verifier, and every one of them can have the same disease. The critic learns what a “good” answer looks like from its training data, and then it rejects good answers that look different. A refactor that takes an unfamiliar approach. A test written in a style the model has seen less of. A correct fix in a language the model is weaker in.
There is also a cross-model version of the question that I have been meaning to test. Would Claude grade an OpenAI model’s output differently from its own? Would the reverse be true? If graders have taste, then a grader from one family may systematically fail work from another, and the pipeline would look like it has quality control when it actually has a preference.
What I am changing
Three things, all of them cheap.
Test the grader on its own. Before I trust a checker in a pipeline, I give it a set of answers I already know are right but written in ways it will not expect. Different structure, different naming, different order. I want to know its false-rejection rate before it starts throwing away good work in production.
Prefer domains with an external verifier. Tests, compilers, type checkers, a script that runs the output. Wherever a mechanical check exists, use it and let the model only handle what the mechanical check can’t. A test suite does not care what the code looks like. That is exactly the property I want.
Learn the scoring function, then sample enough. If there is no mechanical verifier, then the grader is a model I am effectively training with prompts and examples. That means it deserves the same discipline as any model: a held-out set, enough samples to see the error rate, and a rerun whenever I change the prompt.
The lesson from the veto post was that a grader which says yes to everything is a rubber stamp. The lesson from this paper is that a grader which says no to everything unfamiliar is a gatekeeper with a dress code. Neither is a judge. If you want a judge, you have to measure it like one.