Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

TL;DR

This paper introduces a reflective test-time planning approach for embodied LLMs, enabling robots to learn from mistakes through internal and external reflection mechanisms, improving task performance and generalization.

Contribution

It presents a novel framework combining reflection-in-action, reflection-on-action, and retrospective reflection for embodied LLMs, inspired by human practitioners.

Findings

Significant performance improvements on household and MuJoCo benchmarks.

Zero-shot generalization to photorealistic environments and real robot experiments.

Retrospective reflection outperforms step-wise external feedback with less computation.

Abstract

Embodied LLMs endow robots with high-level task reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: \textit{reflection-in-action}, where the agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and \textit{reflection-on-action}, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark show significant gains over baseline models, with zero-shot generalization to photorealistic HM3D environments and real-robot experiments on a Franka Panda arm. Ablations confirm that reflection-in-action and reflection-on-action are mutually dependent, and that retrospective reflection achieves better credit assignment than step-wise external feedback at lower computational overhead. Qualitative analyses further highlight behavioral correction through reflection.