SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval

TL;DR

SubSearch introduces intermediate intrinsic rewards to guide unsupervised reasoning in large language models, improving multi-step reasoning and robustness in complex retrieval tasks without external supervision.

Contribution

It proposes a novel framework that directly optimizes reasoning processes with intrinsic rewards, reducing reliance on annotated trajectories and enhancing autonomous reasoning capabilities.

Findings

Rewarded intermediate reasoning steps improve robustness in QA tasks.

SubSearch outperforms outcome-only reward models on seven benchmarks.

Intrinsic rewards enable autonomous, data-efficient reasoning without external supervision.

Abstract

Large language models (LLMs) are probabilistic in nature and perform more reliably when augmented with external information. As complex queries often require multi-step reasoning over the retrieved information, with no clear or predetermined reasoning path, they remain challenging. Recent approaches train models using reinforcement learning on the model's outcome, showing promise in improving how models handle complex information. We introduce SubSearch, a specialized framework that shifts from outcome-only supervision to intermediate reward signals that incentivize planning high-quality reasoning. Unlike previous work on process reward modeling, which focuses on training a separate reward model with annotated trajectories by either human annotators or large LLM judges, SubSearch directly optimizes the generator using intrinsic process rewards, which we define as internally-derived rewards, eliminating the need for external supervision, and moving towards autonomous information-intensive reasoning. Experiments on seven benchmarks show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces in both QA and multi-hop QA datasets over using only outcome rewards. SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering, while offering a data-efficient alternative to supervised process modeling.