When Neural Networks Using Different Sensors Create Similar Features

TL;DR

This paper investigates how neural networks trained on different sensors for multimodal tasks extract features that are fundamentally similar, revealing that the most correlated features across sensors align with the classification components.

Contribution

It provides an analysis showing that features from different sensor-based neural networks contain similar information, linking feature correlation to classification components.

Findings

Features from different sensors are highly correlated.

Correlated features correspond to classification components.

Neural networks from various sensors extract similar high-level information.

Abstract

Multimodal problems are omnipresent in the real world: autonomous driving, robotic grasping, scene understanding, etc... We draw from the well-developed analysis of similarity to provide an example of a problem where neural networks are trained from different sensors, and where the features extracted from these sensors still carry similar information. More precisely, we demonstrate that for each sensor, the linear combination of the features from the last layer that correlates the most with other sensors corresponds to the classification components of the classification layer.