TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs

TL;DR

TIPS introduces a turn-level reward shaping method for search-augmented LLMs, significantly improving training stability and performance on QA benchmarks by providing dense, fine-grained rewards based on increased answer likelihood.

Contribution

The paper proposes TIPS, a novel turn-level reward shaping framework that enhances reinforcement learning for LLMs by addressing sparse rewards and credit assignment issues.

Findings

TIPS outperforms PPO baselines on seven QA benchmarks.

TIPS improves Exact Match scores by 11.8% and F1 by 13.6% on average.

Training stability is substantially enhanced with TIPS.

Abstract

Search-augmented large language models (LLMs) trained with reinforcement learning (RL) have achieved strong results on open-domain question answering (QA), but training still remains a significant challenge. The optimization is often unstable due to sparse rewards and difficult credit assignments across reasoning and tool calls. To address this, we introduce Turn-Level Information Potential Reward Shaping (TIPS), a simple framework that assigns dense, turn-level rewards to each reasoning + tool-call segment based on the increased likelihood of the correct answer under a teacher model. By leveraging the potential-based reward shaping, TIPS offers fine-grained and policy-invariant guidance that overcomes the limitations of outcome-only optimization. Evaluated on seven QA benchmarks, TIPS consistently outperforms GRPO/PPO baselines and substantially improves training stability. For instance, with a Qwen-2.5 7B Instruct model, TIPS improves the average Exact Match score by 11.8% and F1 by 13.6% relative to PPO. Our results demonstrate that turn-level information-potential reward shaping provides an effective and general solution to sparse-reward credit assignment for multi-turn LLM reasoning.