# Robust Multi-Modal Sensor Fusion: An Adversarial Approach

## TL;DR

This paper presents a robust multi-modal sensor fusion method using a generative network to learn a latent space, improving target detection and classification even with damaged or noisy sensors.

## Contribution

It introduces a data-driven fusion approach that enhances robustness by exploiting an estimated latent space conditioned on sensor modalities, allowing flexibility and damage detection.

## Key findings

- Achieves robustness against noisy and damaged sensors.
- Improves detection and classification performance.
- Demonstrates effectiveness in unconstrained surveillance settings.

## Abstract

In recent years, multi-modal fusion has attracted a lot of research interest, both in academia, and in industry. Multimodal fusion entails the combination of information from a set of different types of sensors. Exploiting complementary information from different sensors, we show that target detection and classification problems can greatly benefit from this fusion approach and result in a performance increase. To achieve this gain, the information fusion from various sensors is shown to require some principled strategy to ensure that additional information is constructively used, and has a positive impact on performance. We subsequently demonstrate the viability of the proposed fusion approach by weakening the strong dependence on the functionality of all sensors, hence introducing additional flexibility in our solution and lifting the severe limitation in unconstrained surveillance settings with potential environmental impact. Our proposed data driven approach to multimodal fusion, exploits selected optimal features from an estimated latent space of data across all modalities. This hidden space is learned via a generative network conditioned on individual sensor modalities. The hidden space, as an intrinsic structure, is then exploited in detecting damaged sensors, and in subsequently safeguarding the performance of the fused sensor system. Experimental results show that such an approach can achieve automatic system robustness against noisy/damaged sensors.