Teaching Robots Novel Objects by Pointing at Them

TL;DR

This paper introduces a neural network-based method enabling robots to learn about new objects through human pointing gestures, allowing for real-time identification and manipulation of previously unseen objects in various environments.

Contribution

The paper presents a novel spatial attention mechanism that helps robots focus on objects indicated by pointing, facilitating learning and manipulation of novel objects during operation.

Findings

Effective object localization in synthetic and real-world datasets.

Robust manipulation of novel objects based on human pointing cues.

Improved robot learning in dynamic, unstructured environments.

Abstract

Robots that must operate in novel environments and collaborate with humans must be capable of acquiring new knowledge from human experts during operation. We propose teaching a robot novel objects it has not encountered before by pointing a hand at the new object of interest. An end-to-end neural network is used to attend to the novel object of interest indicated by the pointing hand and then to localize the object in new scenes. In order to attend to the novel object indicated by the pointing hand, we propose a spatial attention modulation mechanism that learns to focus on the highlighted object while ignoring the other objects in the scene. We show that a robot arm can manipulate novel objects that are highlighted by pointing a hand at them. We also evaluate the performance of the proposed architecture on a synthetic dataset constructed using emojis and on a real-world dataset of common objects.