ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion

TL;DR

This paper introduces ADMC, an attention-based diffusion model that effectively completes missing modality features in multimodal emotion and intent recognition, improving accuracy in both missing and complete data scenarios.

Contribution

The paper presents a novel attention-based diffusion framework that independently trains modality-specific feature extractors and generates missing features, avoiding over-coupling and improving multimodal recognition.

Findings

Achieves state-of-the-art results on IEMOCAP and MIntRec benchmarks.

Effectively handles various missing modality scenarios.

Enhances recognition accuracy even with complete modalities.

Abstract

Multimodal emotion and intent recognition is essential for automated human-computer interaction, It aims to analyze users' speech, text, and visual information to predict their emotions or intent. One of the significant challenges is that missing modalities due to sensor malfunctions or incomplete data. Traditional methods that attempt to reconstruct missing information often suffer from over-coupling and imprecise generation processes, leading to suboptimal outcomes. To address these issues, we introduce an Attention-based Diffusion model for Missing Modalities feature Completion (ADMC). Our framework independently trains feature extraction networks for each modality, preserving their unique characteristics and avoiding over-coupling. The Attention-based Diffusion Network (ADN) generates missing modality features that closely align with authentic multimodal distribution, enhancing performance across all missing-modality scenarios. Moreover, ADN's cross-modal generation offers improved recognition even in full-modality contexts. Our approach achieves state-of-the-art results on the IEMOCAP and MIntRec benchmarks, demonstrating its effectiveness in both missing and complete modality scenarios.