Multi-Level Compositional Reasoning for Interactive Instruction Following

TL;DR

This paper introduces a multi-level reasoning framework for robotic agents to perform complex domestic tasks by breaking down instructions into subgoals, improving efficiency without relying on rule-based planning.

Contribution

It presents the MCR-Agent, a three-level policy system that infers subgoals, controls navigation, and executes manipulation, advancing interactive instruction understanding.

Findings

Achieves 2.03% improvement in efficiency metric (PLWSR) on unseen tasks.

Generates human-interpretable subgoals for complex tasks.

Does not rely on rule-based planning or semantic spatial memory.

Abstract

Robotic agents performing domestic chores by natural language directives are required to master the complex job of navigating environment and interacting with objects in the environments. The tasks given to the agents are often composite thus are challenging as completing them require to reason about multiple subtasks, e.g., bring a cup of coffee. To address the challenge, we propose to divide and conquer it by breaking the task into multiple subgoals and attend to them individually for better navigation and interaction. We call it Multi-level Compositional Reasoning Agent (MCR-Agent). Specifically, we learn a three-level action policy. At the highest level, we infer a sequence of human-interpretable subgoals to be executed based on language instructions by a high-level policy composition controller. At the middle level, we discriminatively control the agent's navigation by a master policy by alternating between a navigation policy and various independent interaction policies. Finally, at the lowest level, we infer manipulation actions with the corresponding object masks using the appropriate interaction policy. Our approach not only generates human interpretable subgoals but also achieves 2.03% absolute gain to comparable state of the arts in the efficiency metric (PLWSR in unseen set) without using rule-based planning or a semantic spatial memory.