A Multisensory Learning Architecture for Rotation-invariant Object Recognition

TL;DR

This paper introduces a multisensory learning architecture utilizing color and depth data from a robot to achieve rotation-invariant object recognition, demonstrating improved accuracy over single-modality and traditional fusion methods.

Contribution

The study proposes a novel multisensory architecture combining CNNs and MLPs for joint feature learning from color and depth data, with a new dataset from the iCub robot.

Findings

Architecture outperforms single-modality models.

Improves recognition accuracy over decision-level fusion.

Validates effectiveness with a new multisensory dataset.

Abstract

This study presents a multisensory machine learning architecture for object recognition by employing a novel dataset that was constructed with the iCub robot, which is equipped with three cameras and a depth sensor. The proposed architecture combines convolutional neural networks to form representations (i.e., features) for grayscaled color images and a multi-layer perceptron algorithm to process depth data. To this end, we aimed to learn joint representations of different modalities (e.g., color and depth) and employ them for recognizing objects. We evaluate the performance of the proposed architecture by benchmarking the results obtained with the models trained separately with the input of different sensors and a state-of-the-art data fusion technique, namely decision level fusion. The results show that our architecture improves the recognition accuracy compared with the models that use inputs from a single modality and decision level multimodal fusion method.