Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning

TL;DR

This paper introduces a Discrepancy-Aware Attention Network that improves audio-visual zero-shot learning by addressing modality quality and content discrepancies, achieving state-of-the-art results.

Contribution

It proposes novel modules, QDMA and CSGM, to mitigate quality and content discrepancies in audio-visual ZSL, enhancing discriminative capabilities.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Validates effectiveness of proposed modules through ablation studies.

Balances modality contributions for improved zero-shot learning.

Abstract

Audio-visual Zero-Shot Learning (ZSL) has attracted significant attention for its ability to identify unseen classes and perform well in video classification tasks. However, modal imbalance in (G)ZSL leads to over-reliance on the optimal modality, reducing discriminative capabilities for unseen classes. Some studies have attempted to address this issue by modifying parameter gradients, but two challenges still remain: (a) Quality discrepancies, where modalities offer differing quantities and qualities of information for the same concept. (b) Content discrepancies, where sample contributions within a modality vary significantly. To address these challenges, we propose a Discrepancy-Aware Attention Network (DAAN) for Enhanced Audio-Visual ZSL. Our approach introduces a Quality-Discrepancy Mitigation Attention (QDMA) unit to minimize redundant information in the high-quality modality and a Contrastive Sample-level Gradient Modulation (CSGM) block to adjust gradient magnitudes and balance content discrepancies. We quantify modality contributions by integrating optimization and convergence rate for more precise gradient modulation in CSGM. Experiments demonstrates DAAN achieves state-of-the-art performance on benchmark datasets, with ablation studies validating the effectiveness of individual modules.