Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following

TL;DR

This paper introduces a robust, closed-loop few-shot planner for Embodied Instruction Following using LLMs, which adapts dynamically and outperforms traditional supervised methods on the ALFRED dataset.

Contribution

It proposes a novel hindsight-based, closed-loop planning approach with an adaptation module for EIF, improving robustness and few-shot performance.

Findings

Achieves competitive performance with few-shot learning.

Surpasses full-shot supervised agent performance.

Demonstrates robustness in out-of-distribution states.

Abstract

This work focuses on building a task planner for Embodied Instruction Following (EIF) using Large Language Models (LLMs). Previous works typically train a planner to imitate expert trajectories, treating this as a supervised task. While these methods achieve competitive performance, they often lack sufficient robustness. When a suboptimal action is taken, the planner may encounter an out-of-distribution state, which can lead to task failure. In contrast, we frame the task as a Partially Observable Markov Decision Process (POMDP) and aim to develop a robust planner under a few-shot assumption. Thus, we propose a closed-loop planner with an adaptation module and a novel hindsight method, aiming to use as much information as possible to assist the planner. Our experiments on the ALFRED dataset indicate that our planner achieves competitive performance under a few-shot assumption. For the first time, our few-shot agent's performance approaches and even surpasses that of the full-shot supervised agent.