Co-Evolution of Policy and Internal Reward for Language Agents

TL;DR

This paper introduces Self-Guide, an internal reward mechanism for language agents that enhances both inference guidance and policy training, leading to significant performance improvements.

Contribution

It proposes a co-evolving loop where internal reward and policy improve together, enabling better long-horizon learning for language agents.

Findings

Inference-time self-guidance improves performance.

Joint evolution of policy and internal reward yields 8% gains.

Internal reward learning enhances long-term policy optimization.

Abstract

Large language model (LLM) agents learn by interacting with environments, but long-horizon training remains fundamentally bottlenecked by sparse and delayed rewards. Existing methods typically address this challenge through post-hoc credit assignment or external reward models, which provide limited guidance at inference time and often separate reward improvement from policy improvement. We propose Self-Guide, a self-generated internal reward for language agents that supports both inference-time guidance and training-time supervision. Specifically, the agent uses Self-Guide as a short self-guidance signal to steer the next action during inference, and converts the same signal into step-level internal reward for denser policy optimization during training. This creates a co-evolving loop: better policy produces better guidance, and better guidance further improves policy as internal reward. Across three agent benchmarks, inference-time self-guidance already yields clear gains, while jointly evolving policy and internal reward with GRPO brings further improvements (8\%) over baselines trained solely with environment reward. Overall, our results suggest that language agents can improve not only by collecting more experience, but also by learning to generate and refine their own internal reward during acting and learning.