Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents

TL;DR

This paper introduces extpolicy, a lightweight fine-tuning method that enhances small language models' ability to reliably retrieve and generate evidence-grounded answers, significantly improving their performance on complex reasoning tasks.

Contribution

The paper presents extpolicy, a novel fine-tuning approach that enables small language models to effectively utilize search tools, matching large language models' performance on reasoning benchmarks.

Findings

extpolicy improves SLM performance by 17.3 on Bamboogle and 15.3 on HotpotQA.

SLMs invoke search tools less frequently and are more prone to hallucinations without extpolicy.

Adaptive search strategies in SLMs often degrade performance, emphasizing the need for consistent search behavior.

Abstract

Agents equipped with search tools have emerged as effective solutions for knowledge-intensive tasks. While Large Language Models (LLMs) exhibit strong reasoning capabilities, their high computational cost limits practical deployment for search agents. Consequently, recent work has focused on distilling agentic behaviors from LLMs into Small Language Models (SLMs). Through comprehensive evaluation on complex multi-hop reasoning tasks, we find that despite possessing less parametric knowledge, SLMs invoke search tools less frequently and are more prone to hallucinations. To address this issue, we propose \policy, a lightweight fine-tuning approach that explicitly trains SLMs to reliably retrieve and generate answers grounded in retrieved evidence. Compared to agent distillation from LLMs, our approach improves performance by 17.3 scores on Bamboogle and 15.3 scores on HotpotQA, achieving LLM-level results across benchmarks. Our further analysis reveals that adaptive search strategies in SLMs often degrade performance, highlighting the necessity of consistent search behavior for reliable reasoning.