Clarifying the Half Full or Half Empty Question: Multimodal Container Classification

TL;DR

This paper evaluates multimodal data fusion techniques for robotic container classification, demonstrating that combining visual, tactile, and proprioceptive data significantly improves accuracy over single-modality approaches.

Contribution

It compares and analyzes three different multimodal fusion strategies in a robotic context, highlighting the advantages of multimodal integration for classification tasks.

Findings

Multimodal fusion improves classification accuracy by 15%.

Different fusion strategies have varying effectiveness depending on data timing.

Multimodal integration outperforms single-sense approaches in robotic perception.

Abstract

Multimodal integration is a key component of allowing robots to perceive the world. Multimodality comes with multiple challenges that have to be considered, such as how to integrate and fuse the data. In this paper, we compare different possibilities of fusing visual, tactile and proprioceptive data. The data is directly recorded on the NICOL robot in an experimental setup in which the robot has to classify containers and their content. Due to the different nature of the containers, the use of the modalities can wildly differ between the classes. We demonstrate the superiority of multimodal solutions in this use case and evaluate three fusion strategies that integrate the data at different time steps. We find that the accuracy of the best fusion strategy is 15% higher than the best strategy using only one singular sense.