BusyBot: Learning to Interact, Reason, and Plan in a BusyBoard Environment

TL;DR

BusyBot is a comprehensive learning framework that enables a robot to interact, reason about, and plan in complex environments with diverse objects, demonstrating generalization to unseen scenarios in both simulation and real-world settings.

Contribution

The paper introduces BusyBoard, a new environment for rich visual feedback, and BusyBot, a self-supervised framework for integrated interaction, reasoning, and planning in robotic manipulation.

Findings

BusyBot effectively learns interaction policies in complex environments.

It successfully reasons about inter-object relations through causal discovery.

The approach generalizes to unseen objects and relations in real-world tests.

Abstract

We introduce BusyBoard, a toy-inspired robot learning environment that leverages a diverse set of articulated objects and inter-object functional relations to provide rich visual feedback for robot interactions. Based on this environment, we introduce a learning framework, BusyBot, which allows an agent to jointly acquire three fundamental capabilities (interaction, reasoning, and planning) in an integrated and self-supervised manner. With the rich sensory feedback provided by BusyBoard, BusyBot first learns a policy to efficiently interact with the environment; then with data collected using the policy, BusyBot reasons the inter-object functional relations through a causal discovery network; and finally by combining the learned interaction policy and relation reasoning skill, the agent is able to perform goal-conditioned manipulation tasks. We evaluate BusyBot in both simulated and real-world environments, and validate its generalizability to unseen objects and relations. Video is available at https://youtu.be/EJ98xBJZ9ek.