Embodied Concept Learner: Self-supervised Learning of Concepts and Mapping through Instruction Following

TL;DR

The Embodied Concept Learner (ECL) enables robots to learn visual concepts and depth perception through interaction and language instructions without supervision, improving instruction following and task reasoning in 3D environments.

Contribution

ECL introduces a modular, self-supervised framework for grounding concepts, constructing semantic maps, and executing tasks from language, with enhanced interpretability and reusability.

Findings

Outperforms previous methods on the ALFRED benchmark.

Learns semantics and depth unsupervisedly through interaction.

Provides transparent, step-by-step planning process.

Abstract

Humans, even at a very early age, can learn visual concepts and understand geometry and layout through active interaction with the environment, and generalize their compositions to complete tasks described by natural languages in novel scenes. To mimic such capability, we propose Embodied Concept Learner (ECL) in an interactive 3D environment. Specifically, a robot agent can ground visual concepts, build semantic maps and plan actions to complete tasks by learning purely from human demonstrations and language instructions, without access to ground-truth semantic and depth supervisions from simulations. ECL consists of: (i) an instruction parser that translates the natural languages into executable programs; (ii) an embodied concept learner that grounds visual concepts based on language descriptions; (iii) a map constructor that estimates depth and constructs semantic maps by leveraging the learned concepts; and (iv) a program executor with deterministic policies to execute each program. ECL has several appealing benefits thanks to its modularized design. Firstly, it enables the robotic agent to learn semantics and depth unsupervisedly acting like babies, e.g., ground concepts through active interaction and perceive depth by disparities when moving forward. Secondly, ECL is fully transparent and step-by-step interpretable in long-term planning. Thirdly, ECL could be beneficial for the embodied instruction following (EIF), outperforming previous works on the ALFRED benchmark when the semantic label is not provided. Also, the learned concept can be reused for other downstream tasks, such as reasoning of object states. Project page: http://ecl.csail.mit.edu/