HAPFI: History-Aware Planning based on Fused Information

TL;DR

HAPFI introduces a novel approach that effectively fuses multi-modal historical data to improve long-term planning in embodied instruction following tasks, demonstrating superior action planning and re-planning capabilities.

Contribution

The paper presents HAPFI, a new method that integrates diverse historical multi-modal information using a Mutually Attentive Fusion technique for enhanced decision-making.

Findings

HAPFI outperforms methods ignoring historical data in planning tasks.

Multi-modal historical data improves re-planning after failures.

Qualitative analysis shows robust re-planning with HAPFI.

Abstract

Embodied Instruction Following (EIF) is a task of planning a long sequence of sub-goals given high-level natural language instructions, such as "Rinse a slice of lettuce and place on the white table next to the fork". To successfully execute these long-term horizon tasks, we argue that an agent must consider its past, i.e., historical data, when making decisions in each step. Nevertheless, recent approaches in EIF often neglects the knowledge from historical data and also do not effectively utilize information across the modalities. To this end, we propose History-Aware Planning based on Fused Information (HAPFI), effectively leveraging the historical data from diverse modalities that agents collect while interacting with the environment. Specifically, HAPFI integrates multiple modalities, including historical RGB observations, bounding boxes, sub-goals, and high-level instructions, by effectively fusing modalities via our Mutually Attentive Fusion method. Through experiments with diverse comparisons, we show that an agent utilizing historical multi-modal information surpasses all the compared methods that neglect the historical data in terms of action planning capability, enabling the generation of well-informed action plans for the next step. Moreover, we provided qualitative evidence highlighting the significance of leveraging historical multi-modal data, particularly in scenarios where the agent encounters intermediate failures, showcasing its robust re-planning capabilities.