Internalizing Agency from Reflective Experience

TL;DR

This paper introduces LEAFE, a framework that enhances autonomous language models' problem-solving ability by internalizing recovery strategies from reflective experience, leading to improved long-horizon interaction performance.

Contribution

LEAFE is the first method to internalize recovery agency from reflective experience, enabling models to better utilize environment feedback for long-term problem solving.

Findings

LEAFE improves Pass@1 by up to 14% over base models.

LEAFE outperforms outcome-driven and experience-based baselines in diverse tasks.

LEAFE enhances the model's ability to recover and explore effectively in complex environments.

Abstract

Large language models are increasingly deployed as autonomous agents that must plan, act, and recover from mistakes through long-horizon interaction with environments that provide rich feedback. However, prevailing outcome-driven post-training methods (e.g., RL with verifiable rewards) primarily optimize final success signals, leaving rich environment feedback underutilized. Consequently, they often lead to distribution sharpening: the policy becomes better at reproducing a narrow set of already-successful behaviors, while failing to improve the feedback-grounded agency needed to expand problem-solving capacity (e.g., Pass@k) in long-horizon settings. To address this, we propose LEAFE (Learning Feedback-Grounded Agency from Reflective Experience), a framework that internalizes recovery agency from reflective experience. Specifically, during exploration, the agent summarizes environment feedback into actionable experience, backtracks to earlier decision points, and explores alternative branches with revised actions. We then distill these experience-guided corrections into the model through supervised fine-tuning, enabling the policy to recover more effectively in future interactions. Across a diverse set of interactive coding and agentic tasks under fixed interaction budgets, LEAFE consistently improves Pass@1 over the base model and achieves higher Pass@k than outcome-driven baselines (GRPO) and experience-based methods such as Early Experience, with gains of up to 14% on Pass@128.