Layout-aware Dreamer for Embodied Referring Expression Grounding

TL;DR

This paper introduces Layout-aware Dreamer, a novel agent that uses layout inference and goal imagination to improve navigation and object grounding in unseen environments, achieving state-of-the-art results.

Contribution

The paper proposes the Layout-aware Dreamer with two modules, Layout Learner and Goal Dreamer, to incorporate environmental layout understanding and goal imagination into embodied referring expression grounding.

Findings

Achieves new state-of-the-art on REVERIE dataset

Improves navigation success rate by 4.02%

Enhances remote grounding success by 3.43%

Abstract

In this work, we study the problem of Embodied Referring Expression Grounding, where an agent needs to navigate in a previously unseen environment and localize a remote object described by a concise high-level natural language instruction. When facing such a situation, a human tends to imagine what the destination may look like and to explore the environment based on prior knowledge of the environmental layout, such as the fact that a bathroom is more likely to be found near a bedroom than a kitchen. We have designed an autonomous agent called Layout-aware Dreamer (LAD), including two novel modules, that is, the Layout Learner and the Goal Dreamer to mimic this cognitive decision process. The Layout Learner learns to infer the room category distribution of neighboring unexplored areas along the path for coarse layout estimation, which effectively introduces layout common sense of room-to-room transitions to our agent. To learn an effective exploration of the environment, the Goal Dreamer imagines the destination beforehand. Our agent achieves new state-of-the-art performance on the public leaderboard of the REVERIE dataset in challenging unseen test environments with improvement in navigation success (SR) by 4.02% and remote grounding success (RGS) by 3.43% compared to the previous state-of-the-art. The code is released at https://github.com/zehao-wang/LAD