Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos

TL;DR

This paper introduces a multimodal approach for stereo sound event localization and detection in videos, integrating semantic embeddings and autocorrelation features to improve accuracy beyond traditional methods.

Contribution

The authors develop a novel Cross-Modal Conformer architecture that fuses audio, visual, and semantic embeddings, enhancing SELD performance with pre-training and data augmentation techniques.

Findings

Significant performance improvement over baseline models.

Effective integration of semantic and visual information.

Enhanced distance estimation with autocorrelation features.

Abstract

This report presents our systems submitted to the audio-only and audio-visual tracks of the DCASE2025 Task 3 Challenge: Stereo Sound Event Localization and Detection (SELD) in Regular Video Content. SELD is a complex task that combines temporal event classification with spatial localization, requiring reasoning across spatial, temporal, and semantic dimensions. The last is arguably the most challenging to model. Traditional SELD architectures rely on multichannel input, which limits their ability to leverage large-scale pre-training due to data constraints. To address this, we enhance standard SELD architectures with semantic information by integrating pre-trained, contrastive language-aligned models: CLAP for audio and OWL-ViT for visual inputs. These embeddings are incorporated into a modified Conformer module tailored for multimodal fusion, which we refer to as the Cross-Modal Conformer. Additionally, we incorporate autocorrelation-based acoustic features to improve distance estimation. We pre-train our models on curated synthetic audio and audio-visual datasets and apply a left-right channel swapping augmentation to further increase the training data. Both our audio-only and audio-visual systems substantially outperform the challenge baselines on the development set, demonstrating the effectiveness of our strategy. Performance is further improved through model ensembling and a visual post-processing step based on human keypoints. Future work will investigate the contribution of each modality and explore architectural variants to further enhance results.