A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition

TL;DR

This paper introduces a robust multimodal person recognition framework that effectively handles missing data from audio, visible, and thermal sensors by using a novel deep latent embedding and attention-based fusion, improving accuracy.

Contribution

The paper presents a new trimodal sensor fusion framework with a novel deep latent embedding and missing modality loss, addressing the challenge of incomplete sensor data in person recognition.

Findings

Significantly improves recognition accuracy with missing modalities.

Outperforms baseline algorithms on the Speaking Faces dataset.

Effectively learns from incomplete multimodal data using the proposed methods.

Abstract

Utilizing the sensor characteristics of the audio, visible camera, and thermal camera, the robustness of person recognition can be enhanced. Existing multimodal person recognition frameworks are primarily formulated assuming that multimodal data is always available. In this paper, we propose a novel trimodal sensor fusion framework using the audio, visible, and thermal camera, which addresses the missing modality problem. In the framework, a novel deep latent embedding framework, termed the AVTNet, is proposed to learn multiple latent embeddings. Also, a novel loss function, termed missing modality loss, accounts for possible missing modalities based on the triplet loss calculation while learning the individual latent embeddings. Additionally, a joint latent embedding utilizing the trimodal data is learnt using the multi-head attention transformer, which assigns attention weights to the different modalities. The different latent embeddings are subsequently used to train a deep neural network. The proposed framework is validated on the Speaking Faces dataset. A comparative analysis with baseline algorithms shows that the proposed framework significantly increases the person recognition accuracy while accounting for missing modalities.