Daily Digest
Test review against the better model
A reviewer model can cost more and score lower. Test it fairly.
The problem
A common way to make AI output more reliable is to add a reviewer: one model answers, a second model checks the answer and fixes it. The usual scorecard compares the reviewed answer with the first model's answer. That number almost always looks good. But if the reviewer is the stronger model, part of that gain is just the stronger model's own skill. It may have done better answering alone, for less money.
The technique
Before you keep a review step, score it against the stronger of the two models working alone, not only against the model it reviews. A study posted Oct 11 (Zhang and Shrestha, arXiv:2610.15897) ran 48 pairs of 12 open models, from 3B to 72B, on math, knowledge, code and reasoning. Review beat the stronger model alone in only 9 of 96 setups. It cost 2.6 to 2.8 times as much. Those are their numbers. The authors' explanation: reviewers mostly go along with the answer they are shown, even when their own answer would have been right.
- Run each model alone on the same questions and write down the better score.
- Run your review step and compare it with that number.
- Also try picking between the two independent answers. The paper's picker was a small classifier, trained on questions with known answers, that used signals like confidence, answer length and whether the two answers agree. It scored 2.7 to 4.2 points above review for less cost.
- Keep review only if it beats the better solo model by enough to pay for the extra call.
What we saw
Our own check was much smaller: two small open models, Qwen2.5 0.5B and 1.5B, on 100 ARC-Challenge science questions, one greedy run each.
Alone, the small model got 28% and the big model got 70%. When the big model reviewed the small model's answers, the final score was 62%. Against the small model that is a 34-point gain. Against the big model alone it is 8 points worse, at about 1.6 times the compute. On the 50 questions where the big model was right alone and the small one was wrong, review ended right only 31 times. In the other direction, the small model reviewing the big one dropped the score from 70% to 46%.
Our picker took whichever answer its model was more confident in. It scored 69%, one point under the big model alone, at 1.3 times the cost. It didn't beat the stronger model either, but it lost far less than review did.
Limits
Our test is small: 100 questions, two small models from one family, one prompt, one run. The 8-point drop points the same way as the paper, but it isn't statistically solid at this size (95% range roughly minus 18 to plus 3 points). Our picker was simpler than the paper's. With bigger models, closer in skill, or tasks where checking is easier than solving, such as running code tests, review can still pay off. The paper found review does worse the bigger the skill gap between the two models, so closely matched pairs are where it has the best chance.
Try it
Take 100 real questions from your workload with known answers. Score model A alone, model B alone, and your review pipeline. If review doesn't beat the better solo score, drop it or swap it for a picker, and keep the savings.