HDA-SELD: Hierarchical Cross-Modal Distillation with Multi-Level Data Augmentation for Low-Resource Audio-Visual Sound Event Localization and Detection

TL;DR

HDA-SELD introduces a hierarchical cross-modal distillation framework with multi-level data augmentation to improve low-resource audio-visual sound event localization and detection, achieving state-of-the-art results.

Contribution

The paper proposes a novel hierarchical cross-modal distillation method combined with multi-level data augmentation for low-resource AV SELD, outperforming existing methods.

Findings

Significant performance improvements on DCASE datasets (21%-38% gains).

Achieves results comparable or superior to larger teacher models.

Surpasses state-of-the-art on DCASE 2023 and 2024 SELD tasks.

Abstract

This work presents HDA-SELD, a unified framework that combines hierarchical cross-modal distillation (HCMD) and multi-level data augmentation to address low-resource audio-visual (AV) sound event localization and detection (SELD). An audio-only SELD model acts as the teacher, transferring knowledge to an AV student model through both output responses and intermediate feature representations. To enhance learning, data augmentation is applied by mixing features randomly selected from multiple network layers and associated loss functions tailored to the SELD task. Extensive experiments on the DCASE 2023 and 2024 Challenge SELD datasets show that the proposed method significantly improves AV SELD performance, yielding relative gains of 21%-38% in the overall metric over the baselines. Notably, our proposed HDA-SELD achieves results comparable to or better than teacher models trained on much larger datasets, surpassing state-of-the-art methods on both DCASE 2023 and 2024 Challenge SELD tasks.