Multi-Level Sensor Fusion with Deep Learning

TL;DR

This paper introduces CentralNet, a deep learning architecture for sensor data fusion that adaptively balances early and late fusion, achieving state-of-the-art results across multiple multimodal datasets.

Contribution

The paper proposes a novel deep network, CentralNet, with multi-objective regularization for automatic and efficient sensor data fusion at multiple abstraction levels.

Findings

Achieves state-of-the-art performance on four multimodal datasets.

Demonstrates ability to select optimal fusion strategies for different problems.

Outperforms existing fusion methods in accuracy and flexibility.

Abstract

In the context of deep learning, this article presents an original deep network, namely CentralNet, for the fusion of information coming from different sensors. This approach is designed to efficiently and automatically balance the trade-off between early and late fusion (i.e. between the fusion of low-level vs high-level information). More specifically, at each level of abstraction-the different levels of deep networks-uni-modal representations of the data are fed to a central neural network which combines them into a common embedding. In addition, a multi-objective regularization is also introduced, helping to both optimize the central network and the unimodal networks. Experiments on four multimodal datasets not only show state-of-the-art performance, but also demonstrate that CentralNet can actually choose the best possible fusion strategy for a given problem.