InfoPO: Information-Driven Policy Optimization for User-Centric Agents

TL;DR

InfoPO introduces an information-driven policy optimization method that enhances multi-turn interactions in user-centric agents by focusing on valuable information gain, leading to improved performance across diverse tasks.

Contribution

It proposes a novel active uncertainty reduction framework with an information-gain reward for better multi-turn policy learning in user-centric agents.

Findings

Outperforms prompting and RL baselines across tasks.

Demonstrates robustness to user simulator shifts.

Generalizes well to environment-interactive tasks.

Abstract

Real-world user requests to LLM agents are often underspecified. Agents must interact to acquire missing information and make correct downstream decisions. However, current multi-turn GRPO-based methods often rely on trajectory-level reward computation, which leads to credit assignment problems and insufficient advantage signals within rollout groups. A feasible approach is to identify valuable interaction turns at a fine granularity to drive more targeted learning. To address this, we introduce InfoPO (Information-Driven Policy Optimization), which frames multi-turn interaction as a process of active uncertainty reduction and computes an information-gain reward that credits turns whose feedback measurably changes the agent's subsequent action distribution compared to a masked-feedback counterfactual. It then combines this signal with task outcomes via an adaptive variance-gated fusion to identify information importance while maintaining task-oriented goal direction. Across diverse tasks, including intent clarification, collaborative coding, and tool-augmented decision making, InfoPO consistently outperforms prompting and multi-turn RL baselines. It also demonstrates robustness under user simulator shifts and generalizes effectively to environment-interactive tasks. Overall, InfoPO provides a principled and scalable mechanism for optimizing complex agent-user collaboration. Code is available at https://github.com/kfq20/InfoPO.