Query Disambiguation via Answer-Free Context: Doubling Performance on Humanity's Last Exam

TL;DR

This paper shows that rewriting questions to reduce ambiguity using answer-free grounding context significantly improves language model accuracy on benchmark tasks, emphasizing the importance of query formulation.

Contribution

It introduces a method combining answer-free grounding context with query rewriting to enhance language model performance on complex questions.

Findings

Rewriting questions with grounding context improves accuracy from 0.14 to 0.37.

Rewriting and answering are distinct phases; prompting alone cannot recover the gains.

Combining dynamic context construction with query rewriting yields substantial performance improvements.

Abstract

How carefully and unambiguously a question is phrased has a profound impact on the quality of the response, for Language Models (LMs) as well as people. While model capabilities continue to advance, the interplay between grounding context and query formulation remains under-explored. This work investigates how the quality of background grounding information in a model's context window affects accuracy. We find that combining well-grounded dynamic context construction (i.e, RAG) with query rewriting reduces question ambiguity, resulting in significant accuracy gains. Given a user question with associated answer-free grounding context, rewriting the question to reduce ambiguity produces benchmark improvements without changing the answer itself, even compared to prepending that context before the question. Using \texttt{gpt-oss-20b} to rewrite a subset of Humanity's Last Exam using answer-free grounding context improves \texttt{gpt-5-mini} accuracy from 0.14 to 0.37. We demonstrate that this accuracy improvement cannot be fully recovered just through prompting at inference time; rather, distinct rewriting and answering phases are required. Code and data are available at https://github.com/mmajurski/lm-rewrite-uplift