Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents

TL;DR

This paper introduces the PET framework, leveraging large language models to decompose tasks, filter observations, and track progress, significantly improving embodied agent performance on complex instruction-following benchmarks.

Contribution

The PET framework uses LLMs to simplify control problems for embodied agents without fine-tuning, addressing architecture constraints and improving generalization.

Findings

15% improvement over SOTA on AlfWorld benchmark

Effective task decomposition and observation filtering

Enhanced generalization to human goal specifications

Abstract

Pre-trained large language models (LLMs) capture procedural knowledge about the world. Recent work has leveraged LLM's ability to generate abstract plans to simplify challenging control tasks, either by action scoring, or action modeling (fine-tuning). However, the transformer architecture inherits several constraints that make it difficult for the LLM to directly serve as the agent: e.g. limited input lengths, fine-tuning inefficiency, bias from pre-training, and incompatibility with non-text environments. To maintain compatibility with a low-level trainable actor, we propose to instead use the knowledge in LLMs to simplify the control problem, rather than solving it. We propose the Plan, Eliminate, and Track (PET) framework. The Plan module translates a task description into a list of high-level sub-tasks. The Eliminate module masks out irrelevant objects and receptacles from the observation for the current sub-task. Finally, the Track module determines whether the agent has accomplished each sub-task. On the AlfWorld instruction following benchmark, the PET framework leads to a significant 15% improvement over SOTA for generalization to human goal specifications.