Audio-Visual Object Classification for Human-Robot Collaboration

TL;DR

This paper introduces the CORSMAL challenge and dataset to evaluate audio-visual algorithms for estimating physical properties and content classification of containers in human-robot collaboration, supporting contactless interaction.

Contribution

It presents a new benchmark with a real-to-simulation framework for assessing audio-visual object property estimation in human-robot tasks.

Findings

Established performance scores for object property estimation

Provided a dataset for audio-visual human-robot interaction

Demonstrated the impact of estimation errors in simulation

Abstract

Human-robot collaboration requires the contactless estimation of the physical properties of containers manipulated by a person, for example while pouring content in a cup or moving a food box. Acoustic and visual signals can be used to estimate the physical properties of such objects, which may vary substantially in shape, material and size, and also be occluded by the hands of the person. To facilitate comparisons and stimulate progress in solving this problem, we present the CORSMAL challenge and a dataset to assess the performance of the algorithms through a set of well-defined performance scores. The tasks of the challenge are the estimation of the mass, capacity, and dimensions of the object (container), and the classification of the type and amount of its content. A novel feature of the challenge is our real-to-simulation framework for visualising and assessing the impact of estimation errors in human-to-robot handovers.