SonicSense: Object Perception from In-Hand Acoustic Vibration

TL;DR

SonicSense is a comprehensive system that uses in-hand acoustic vibration sensing combined with learning algorithms to enable robots to perceive diverse object properties such as shape, material, and identity, surpassing previous limitations.

Contribution

It introduces a hardware-software framework that significantly broadens object perception capabilities through in-hand acoustic sensing and learning, handling diverse real-world objects.

Findings

Differentiates container inventory status

Predicts heterogeneous materials

Reconstructs 3D object shapes

Abstract

We introduce SonicSense, a holistic design of hardware and software to enable rich robot object perception through in-hand acoustic vibration sensing. While previous studies have shown promising results with acoustic sensing for object perception, current solutions are constrained to a handful of objects with simple geometries and homogeneous materials, single-finger sensing, and mixing training and testing on the same objects. SonicSense enables container inventory status differentiation, heterogeneous material prediction, 3D shape reconstruction, and object re-identification from a diverse set of 83 real-world objects. Our system employs a simple but effective heuristic exploration policy to interact with the objects as well as end-to-end learning-based algorithms to fuse vibration signals to infer object properties. Our framework underscores the significance of in-hand acoustic vibration sensing in advancing robot tactile perception.