Scent of Knowledge: Optimizing Search-Enhanced Reasoning with Information Foraging

TL;DR

This paper introduces InForage, a reinforcement learning framework inspired by Information Foraging Theory, enabling LLMs to perform dynamic, adaptive retrieval for complex reasoning tasks, significantly improving over static retrieval methods.

Contribution

InForage is a novel RL-based approach that formalizes retrieval as an adaptive, information-seeking process, with explicit rewards for retrieval quality, advancing beyond static retrieval strategies.

Findings

InForage outperforms baseline methods on question answering and reasoning tasks.

Constructed a human-guided dataset for training iterative search strategies.

Demonstrated robustness and efficiency in real-world web QA scenarios.

Abstract

Augmenting large language models (LLMs) with external retrieval has become a standard method to address their inherent knowledge cutoff limitations. However, traditional retrieval-augmented generation methods employ static, pre-inference retrieval strategies, making them inadequate for complex tasks involving ambiguous, multi-step, or evolving information needs. Recent advances in test-time scaling techniques have demonstrated significant potential in enabling LLMs to dynamically interact with external tools, motivating the shift toward adaptive inference-time retrieval. Inspired by Information Foraging Theory (IFT), we propose InForage, a reinforcement learning framework that formalizes retrieval-augmented reasoning as a dynamic information-seeking process. Unlike existing approaches, InForage explicitly rewards intermediate retrieval quality, encouraging LLMs to iteratively gather and integrate information through adaptive search behaviors. To facilitate training, we construct a human-guided dataset capturing iterative search and reasoning trajectories for complex, real-world web tasks. Extensive evaluations across general question answering, multi-hop reasoning tasks, and a newly developed real-time web QA dataset demonstrate InForage's superior performance over baseline methods. These results highlight InForage's effectiveness in building robust, adaptive, and efficient reasoning agents.