Learning Agentic Policy from Action Guidance

TL;DR

This paper introduces ActGuide-RL, a method that uses human action data as guidance to improve agentic reinforcement learning in large language models, reducing reliance on costly supervised fine-tuning.

Contribution

It proposes a novel approach that injects action data as plan-style guidance, enabling better exploration and learning in reward-sparse tasks without extensive fine-tuning.

Findings

ActGuide-RL significantly outperforms zero RL on search-agent benchmarks.

It matches the performance of SFT+RL pipelines without cold start.

The method effectively internalizes exploration gains through mixed-policy training.

Abstract

Agentic reinforcement learning (RL) for Large Language Models (LLMs) critically depends on the exploration capability of the base policy, as training signals emerge only within its in-capability region. For tasks where the base policy cannot reach reward states, additional training or external guidance is needed to recover effective learning signals. Rather than relying on costly iterative supervised fine tuning (SFT), we exploit the abundant action data generated in everyday human interactions. We propose \textsc{ActGuide-RL}, which injects action data as plan-style reference guidance, enabling the agentic policy to overcome reachability barriers to reward states. Guided and unguided rollouts are then jointly optimized via mixed-policy training, internalizing the exploration gains back into the unguided policy. Motivated by a theoretical and empirical analysis of the benefit-risk trade-off, we adopt a minimal intervention principle that invokes guidance only as an adaptive fallback, matching task difficulty while minimizing off-policy risk. On search-agent benchmarks, \textsc{ActGuide-RL} substantially improves over zero RL (+10.7 pp on GAIA and +19 pp on XBench with Qwen3-4B), and performs on par with the SFT+RL pipeline without any cold start. This suggests a new paradigm for agentic RL that reduces the reliance on heavy SFT data by using scalable action guidance instead.