DISCO: Embodied Navigation and Interaction via Differentiable Scene Semantics and Dual-level Control

TL;DR

DISCO introduces a novel embodied AI framework that combines differentiable scene semantics and dual-level control to improve navigation and interaction tasks in complex environments, significantly outperforming previous methods.

Contribution

The paper presents DISCO, a new approach that integrates dynamic scene semantics and hierarchical control for embodied agents, advancing the state-of-the-art in mobile manipulation and instruction following.

Findings

DISCO achieves +8.6% success rate improvement in unseen scenes.

It effectively models rich scene semantics for better navigation planning.

The dual-level control enhances task efficiency and accuracy.

Abstract

Building a general-purpose intelligent home-assistant agent skilled in diverse tasks by human commands is a long-term blueprint of embodied AI research, which poses requirements on task planning, environment modeling, and object interaction. In this work, we study primitive mobile manipulations for embodied agents, i.e. how to navigate and interact based on an instructed verb-noun pair. We propose DISCO, which features non-trivial advancements in contextualized scene modeling and efficient controls. In particular, DISCO incorporates differentiable scene representations of rich semantics in object and affordance, which is dynamically learned on the fly and facilitates navigation planning. Besides, we propose dual-level coarse-to-fine action controls leveraging both global and local cues to accomplish mobile manipulation tasks efficiently. DISCO easily integrates into embodied tasks such as embodied instruction following. To validate our approach, we take the ALFRED benchmark of large-scale long-horizon vision-language navigation and interaction tasks as a test bed. In extensive experiments, we make comprehensive evaluations and demonstrate that DISCO outperforms the art by a sizable +8.6% success rate margin in unseen scenes, even without step-by-step instructions. Our code is publicly released at https://github.com/AllenXuuu/DISCO.