The Solution for Temporal Sound Localisation Task of ICCV 1st Perception Test Challenge 2023

TL;DR

This paper presents a multimodal fusion approach using visual and audio features with a multi-scale Transformer to improve temporal sound localization, achieving second-best performance in the ICCV 2023 challenge.

Contribution

It introduces a novel combination of high-quality visual features and audio data with a multi-scale Transformer for enhanced sound localization.

Findings

Achieved a mean average precision of 0.33 in the challenge

Outperformed many existing methods in the ICCV 2023 task

Demonstrated the effectiveness of multimodal fusion in sound localization

Abstract

In this paper, we propose a solution for improving the quality of temporal sound localization. We employ a multimodal fusion approach to combine visual and audio features. High-quality visual features are extracted using a state-of-the-art self-supervised pre-training network, resulting in efficient video feature representations. At the same time, audio features serve as complementary information to help the model better localize the start and end of sounds. The fused features are trained in a multi-scale Transformer for training. In the final test dataset, we achieved a mean average precision (mAP) of 0.33, obtaining the second-best performance in this track.