Multimodal sensor fusion in the latent representation space

TL;DR

This paper introduces a novel two-stage multimodal sensor fusion method using generative models, effective for classification, denoising, and reconstructing subsampled data, advancing fusion techniques in latent spaces.

Contribution

The paper presents a new approach that constructs a generative model from unlabeled data to perform sensor fusion in the latent space, handling subsampling scenarios.

Findings

Effective in multisensory classification

High performance in denoising tasks

Successful recovery from subsampled observations

Abstract

A new method for multimodal sensor fusion is introduced. The technique relies on a two-stage process. In the first stage, a multimodal generative model is constructed from unlabelled training data. In the second stage, the generative model serves as a reconstruction prior and the search manifold for the sensor fusion tasks. The method also handles cases where observations are accessed only via subsampling i.e. compressed sensing. We demonstrate the effectiveness and excellent performance on a range of multimodal fusion experiments such as multisensory classification, denoising, and recovery from subsampled observations.