If several verifiers can score candidate answers, it seems natural to ask which verifier to call next. A cheap early score could resolve an easy question. A harder question could receive more scrutiny. The policy would buy information only when it might change the selected answer.
My verifier-acquisition study tested that idea using released cached scores. The most useful finding was that much of the apparent benefit did not require adaptivity at all. A carefully selected static subset explained most of it. A simpler candidate-shortlisting cascade then beat my adaptive policy on all three tested tasks.
The unit of a call matters
The initial action was a judge applied across all candidate answers for one question. With 100 candidates, “one call” represented 100 candidate–judge evaluations. A policy could save judge calls and still spend heavily on weak candidates.
The proposed adaptive rule estimated whether an unacquired judge could overturn the current selection, discounted redundant information, and divided by cost. It used a fitted statistical surrogate. That sounds tailored to the problem, but a sophisticated acquisition score can still be solving the wrong allocation problem.
The evaluation masked unqueried scores and correctness labels at the policy boundary. This is essential: an offline table makes it easy to leak information that a live policy would not have. Operating points were chosen on calibration data rather than selected from the final test results.
The apparently adaptive gain had a fixed cause
One transfer comparison looked especially encouraging. A trace diagnostic showed that the advantage came from the first judge, chosen independently of the question, rather than from the subsequent sequence of decisions.
That changes the interpretation. The method had found a useful fixed ordering for that transfer. Calling the gain evidence for adaptive reasoning would assign the credit to a mechanism the experiment did not need.
Across cost-matched comparisons, static development-selected subsets and confidence stopping explained most of the adaptive policy’s performance. The result did not say additional verification is useless. It said that this policy, at this action granularity, had not shown the claimed value over simpler allocation.
Spend on candidates that can still win
The next experiment moved the budget inside the candidate set. Run one judge on all answers, retain its top candidates, then apply the remaining judges only to that shortlist. This cascade beat the adaptive policy on MATH500, GPQA, and MMLU-Pro with fewer candidate–judge evaluations in the reported setting.
That is a useful reversal of the original design intuition. The important economy came from avoiding repeated scrutiny of candidates that were already unlikely to be selected. It did not require a more elaborate policy for choosing the next judge over the entire pool.
There are limits. A shortlist can discard the correct answer early. The strongest full ensemble on one task still did better. Cached score counts also do not establish live latency, energy, or throughput: batching and model loading can change those relationships.
Before learning a compute policy, locate the headroom
My separate Next-Concept Model study encountered a related issue. On the tested Sudoku checkpoints, no intermediate level produced a complete valid board before the final level across 6,000 board–setting pairs. A clever early-stopping rule could not exploit complete early solutions that were absent.
Selective additional full-depth samples did help when the final board failed an available verifier. That was a different allocation opportunity. The representation supported useful refinement after a full answer, rather than useful termination before one.
These experiments suggest a practical order: measure what an oracle allocation could save, identify the unit where waste occurs, and only then train or design a policy. A negative result can tell us to change the action space instead of continuing to improve the controller.