Multimodal Sensory Learning for Real-time, Adaptive Manipulation

TL;DR

This paper presents a multimodal sensory learning framework combining tactile and audio data to enable real-time, adaptive manipulation of objects, especially when visual information is unreliable or unavailable.

Contribution

It introduces a novel sensory fusion approach and a reactive control system that adaptively manipulates objects based on multimodal sensory predictions.

Findings

Effective sensory fusion improves object property estimation.

Reactive control enhances manipulation safety and accuracy.

Multimodal approach outperforms vision-only methods.

Abstract

Adaptive control for real-time manipulation requires quick estimation and prediction of object properties. While robot learning in this area primarily focuses on using vision, many tasks cannot rely on vision due to object occlusion. Here, we formulate a learning framework that uses multimodal sensory fusion of tactile and audio data in order to quickly characterize and predict an object's properties. The predictions are used in a developed reactive controller to adapt the grip on the object to compensate for the predicted inertial forces experienced during motion. Drawing inspiration from how humans interact with objects, we propose an experimental setup from which we can understand how to best utilize different sensory signals and actively interact with and manipulate objects to quickly learn their object properties for safe manipulation.