Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

TL;DR

This paper introduces R$^{2}$ScP, a retrieval-based framework for incomplete audio-visual question answering that improves semantic recovery and robustness over traditional generative imputation methods.

Contribution

It proposes a novel retrieval-based approach with semantic purification and a two-stage training strategy to better handle missing modalities in AVQA.

Findings

R$^{2}$ScP outperforms existing methods in AVQA tasks with incomplete data.

The framework enhances robustness in modal-incomplete scenarios.

Semantic purification reduces noise and improves reasoning accuracy.

Abstract

Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly. However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions. Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features. While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modality-specific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy. To tackle these challenges, we propose R$^{2}$ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery. Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge. To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data. Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources. Extensive experiments demonstrate that R$^{2}$ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.