Learning Perceptual Concepts by Bootstrapping from Human Queries

TL;DR

This paper introduces Perceptual Concept Bootstrapping (PCB), a framework that uses privileged simulator data and minimal human input to efficiently learn perceptual concepts for robots operating in human environments.

Contribution

The paper presents a novel bootstrapping approach that leverages privileged simulator information to reduce human labeling effort in learning perceptual concepts from high-dimensional sensor data.

Findings

PCB outperforms baseline methods in learning spatial concepts.

Learned concepts improve motion planning on a Franka Panda robot.

Framework effectively reduces human labeling requirements.

Abstract

When robots operate in human environments, it's critical that humans can quickly teach them new concepts: object-centric properties of the environment that they care about (e.g. objects near, upright, etc). However, teaching a new perceptual concept from high-dimensional robot sensor data (e.g. point clouds) is demanding, requiring an unrealistic amount of human labels. To address this, we propose a framework called Perceptual Concept Bootstrapping (PCB). First, we leverage the inherently lower-dimensional privileged information, e.g., object poses and bounding boxes, available from a simulator only at training time to rapidly learn a low-dimensional, geometric concept from minimal human input. Second, we treat this low-dimensional concept as an automatic labeler to synthesize a large-scale high-dimensional data set with the simulator. With these two key ideas, PCB alleviates human label burden while still learning perceptual concepts that work with real sensor input where no privileged information is available. We evaluate PCB for learning spatial concepts that describe object state or multi-object relationships, and show it achieves superior performance compared to baseline methods. We also demonstrate the utility of the learned concepts in motion planning tasks on a 7-DoF Franka Panda robot.