A person in a coral sweater frowns at a long paper receipt that unrolls past the edge, its last line torn off.

Daily Digest

Check for cut-off answers

A better prompt can quietly cut off the answer. Count the cut-offs.

Daily Digest · Edited by Osbel Morell

Technique · Oct 13 · 3 min read

The problem

You change a prompt, accuracy goes up, you ship. But the change also alters how long the model talks. If the new prompt gets the model to reason out loud before it answers, the reply can hit your output limit before the answer arrives. Your code receives half a sentence. If your scoring forgives messy output, or only counts the replies that parsed, the dashboard never shows the loss.

The technique

Treat reply length as part of every prompt change, and count cut-off answers next to your main score. A preregistered study posted Oct 9 (arXiv:2610.13287) recommends exactly this after seeing it happen to three prompt optimizers.

Log each reply's token count and why it stopped: finished, or hit the limit.
After any prompt edit, compare old and new on three numbers: the share of replies that hit the limit, the share with no readable answer, and the 90th-percentile length.
Split failures into two piles. Cut off: the reply would have parsed with more room. Format dropped: it never wrote the required shape at all. They need different fixes.
Fix cut-offs by raising the limit or asking for the answer first. Fix dropped formats in the prompt, or with a strict parser and one retry.

What we saw

The authors ran three optimizers (GEPA, MIPROv2 and TextGrad) on Claude Haiku 4.5. On HotpotQA's multi-step questions, the optimized prompts raised exact-match accuracy from 0.193 to between 0.406 and 0.536. But under a 200-token output limit, 13.9% to 47.0% of replies ended before any answer appeared. At 1,024 tokens every reply finished, the loss on other requirements shrank to under one point, and accuracy rose further, to between 0.549 and 0.639. Those are their numbers.

Our own check was much smaller: a small open model (Qwen2.5-1.5B-Instruct) on 24 HotpotQA questions, one run per prompt. We ran no optimizer; we wrote prompts in the style the paper describes. Answer-only replies had a median of 24 tokens and never lost the answer at any limit from 100 to 1,024 tokens. Asking for step-by-step reasoning before the answer pushed the median to 254 tokens. At a 200-token limit, 18 of 24 replies hit the cap and 19 had no readable answer. Raising the limit to 400 recovered all 8 that had been cut off. The other 11 never produced a readable answer, even at 1,024 tokens. Asking for the answer first and the reasoning after cut answers lost to the 200-token limit from 8 to 2.

One surprise: when the "think first" line sat in the system prompt, our model ignored it and stayed short, in two different wordings. Wording swings reply length both ways, so measure it.

Limits

The paper used one task model from one provider, and nearly all of the effect came from one question set. Ours is 24 questions on a small model, one run each, so treat it as a demonstration, not a rate. A small model drops formats far more often than a frontier model, so don't read our 11 of 24 as typical. Raising the limit costs tokens and time. Moving the answer ahead of the reasoning may give back some of the accuracy that reasoning bought, and neither the paper nor we tested that directly.

Try it

Pick one prompt you changed in the last month. Pull its replies from before and after the change, and count how many stopped at the token limit and how many had no readable answer. If the cut-off share rose, raise the limit above the new 90th-percentile length or ask for the answer before the explanation, then check both numbers again.