Show, Don't Tell: Detecting Novel Objects by Watching Human Videos

TL;DR

This paper introduces a self-supervised, language-free object detection system that rapidly trains bespoke detectors from human demonstrations, significantly improving robot recognition of novel objects during tasks.

Contribution

The paper presents a novel 'Show, Don't Tell' paradigm that trains object detectors directly from demonstration images without language prompts, enabling quick adaptation to new objects.

Findings

Outperforms state-of-the-art detection methods

Improves robot task completion rates

Eliminates need for language prompts and prompt engineering

Abstract

How can a robot quickly identify and recognize new objects shown to it during a human demonstration? Existing closed-set object detectors frequently fail at this because the objects are out-of-distribution. While open-set detectors (e.g., VLMs) sometimes succeed, they often require expensive and tedious human-in-the-loop prompt engineering to uniquely recognize novel object instances. In this paper, we present a self-supervised system that eliminates the need for tedious language descriptions and expensive prompt engineering by training a bespoke object detector on an automatically created dataset, supervised by the human demonstration itself. In our approach, "Show, Don't Tell," we show the detector the specific objects of interest during the demonstration, rather than telling the detector about these objects via complex language descriptions. By bypassing language altogether, this paradigm enables us to quickly train bespoke detectors tailored to the relevant objects observed in human task demonstrations. We develop an integrated on-robot system to deploy our "Show, Don't Tell" paradigm of automatic dataset creation and novel object-detection on a real-world robot. Empirical results demonstrate that our pipeline significantly outperforms state-of-the-art detection and recognition methods for manipulated objects, leading to improved task completion for the robot.