SEM: Reinforcement Learning for Search-Efficient Large Language Models

TL;DR

This paper introduces SEM, a reinforcement learning framework that trains large language models to efficiently decide when to search externally, reducing redundant searches while maintaining high answer accuracy.

Contribution

The paper presents a novel post-training reinforcement learning approach, SEM, that explicitly optimizes search behavior in LLMs using a balanced dataset and structured reasoning templates.

Findings

Reduces redundant search operations significantly.

Maintains or improves answer accuracy on benchmarks.

Enhances reasoning efficiency in large language models.

Abstract

Recent advancements in Large Language Models(LLMs) have demonstrated their capabilities not only in reasoning but also in invoking external tools, particularly search engines. However, teaching models to discern when to invoke search and when to rely on their internal knowledge remains a significant challenge. Existing reinforcement learning approaches often lead to redundant search behaviors, resulting in inefficiencies and over-cost. In this paper, we propose SEM, a novel post-training reinforcement learning framework that explicitly trains LLMs to optimize search usage. By constructing a balanced dataset combining MuSiQue and MMLU, we create scenarios where the model must learn to distinguish between questions it can answer directly and those requiring external retrieval. We design a structured reasoning template and employ Group Relative Policy Optimization(GRPO) to post-train the model's search behaviors. Our reward function encourages accurate answering without unnecessary search while promoting effective retrieval when needed. Experimental results demonstrate that our method significantly reduces redundant search operations while maintaining or improving answer accuracy across multiple challenging benchmarks. This framework advances the model's reasoning efficiency and extends its capability to judiciously leverage external knowledge.