ProAct: Agentic Lookahead in Interactive Environments

TL;DR

ProAct introduces a two-stage training framework for LLM agents that enhances long-horizon planning in interactive environments by internalizing lookahead reasoning and refining decision accuracy, leading to superior performance.

Contribution

ProAct's novel combination of Grounded LookAhead Distillation and Monte-Carlo Critic improves planning and decision-making in LLM agents without expensive inference-time search.

Findings

ProAct significantly improves planning accuracy in complex environments.

A 4B parameter model trained with ProAct outperforms open-source baselines.

ProAct demonstrates robust generalization to unseen environments.

Abstract

Existing Large Language Model (LLM) agents struggle in interactive environments requiring long-horizon planning, primarily due to compounding errors when simulating future states. To address this, we propose ProAct, a framework that enables agents to internalize accurate lookahead reasoning through a two-stage training paradigm. First, we introduce Grounded LookAhead Distillation (GLAD), where the agent undergoes supervised fine-tuning on trajectories derived from environment-based search. By compressing complex search trees into concise, causal reasoning chains, the agent learns the logic of foresight without the computational overhead of inference-time search. Second, to further refine decision accuracy, we propose the Monte-Carlo Critic (MC-Critic), a plug-and-play auxiliary value estimator designed to enhance policy-gradient algorithms like PPO and GRPO. By leveraging lightweight environment rollouts to calibrate value estimates, MC-Critic provides a low-variance signal that facilitates stable policy optimization without relying on expensive model-based value approximation. Experiments on both stochastic (e.g., 2048) and deterministic (e.g., Sokoban) environments demonstrate that ProAct significantly improves planning accuracy. Notably, a 4B parameter model trained with ProAct outperforms all open-source baselines and rivals state-of-the-art closed-source models, while demonstrating robust generalization to unseen environments. The codes and models are available at https://github.com/GreatX3/ProAct