Long-horizon Embodied Planning with Implicit Logical Inference and Hallucination Mitigation

TL;DR

ReLEP is a novel framework enabling long-horizon embodied AI planning without in-context examples by learning implicit logical inference and hallucination mitigation through fine-tuning a vision-language model.

Contribution

The paper introduces ReLEP, a fine-tuned vision-language model with a skill library, memory, and configuration modules for versatile, long-horizon embodied planning without relying on in-context examples.

Findings

ReLEP achieves high success rates on various long-horizon tasks.

ReLEP outperforms state-of-the-art baselines.

ReLEP effectively mitigates hallucinations in planning.

Abstract

Long-horizon embodied planning underpins embodied AI. To accomplish long-horizon tasks, one of the most feasible ways is to decompose abstract instructions into a sequence of actionable steps. Foundation models still face logical errors and hallucinations in long-horizon planning, unless provided with highly relevant examples to the tasks. However, providing highly relevant examples for any random task is unpractical. Therefore, we present ReLEP, a novel framework for Real-time Long-horizon Embodied Planning. ReLEP can complete a wide range of long-horizon tasks without in-context examples by learning implicit logical inference through fine-tuning. The fine-tuned large vision-language model formulates plans as sequences of skill functions. These functions are selected from a carefully designed skill library. ReLEP is also equipped with a Memory module for plan and status recall, and a Robot Configuration module for versatility across robot types. In addition, we propose a data generation pipeline to tackle dataset scarcity. When constructing the dataset, we considered the implicit logical relationships, enabling the model to learn implicit logical relationships and dispel hallucinations. Through comprehensive evaluations across various long-horizon tasks, ReLEP demonstrates high success rates and compliance to execution even on unseen tasks and outperforms state-of-the-art baseline methods.