ThinkBot: Embodied Instruction Following with Thought Chain Reasoning

TL;DR

ThinkBot enhances embodied instruction following by reasoning through thought chains to recover missing actions, leading to more successful and efficient task completion in complex environments.

Contribution

It introduces a thought chain reasoning approach with an instruction completer and object localizer, improving EIF performance over existing methods.

Findings

Outperforms state-of-the-art EIF methods in success rate

Achieves higher execution efficiency

Effective in complex simulated environments

Abstract

Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. Conventional methods directly consider the sparse human instruction to generate action plans for agents, which usually fail to achieve human goals because of the instruction incoherence in action descriptions. On the contrary, we propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions, so that the agent can successfully complete human goals by following the coherent instruction. Specifically, we first design an instruction completer based on large language models to recover the missing actions with interacted objects between consecutive human instruction, where the perceived surrounding environments and the completed sub-goals are considered for instruction completion. Based on the partially observed scene semantic maps, we present an object localizer to infer the position of interacted objects for agents to achieve complex human goals. Extensive experiments in the simulated environment show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.