FlowGrad: Using Motion for Visual Sound Source Localization

TL;DR

FlowGrad introduces motion-based optical flow features into visual sound source localization, enhancing performance in complex urban scenes by leveraging temporal information often neglected in prior semantic-focused methods.

Contribution

The paper presents a novel approach that incorporates optical flow to encode motion, improving sound source localization in challenging urban environments.

Findings

Optical flow enhances localization accuracy in urban scenes.

Temporal context improves robustness over static semantic methods.

Analysis reveals strengths and limitations of motion-based localization.

Abstract

Most recent work in visual sound source localization relies on semantic audio-visual representations learned in a self-supervised manner, and by design excludes temporal information present in videos. While it proves to be effective for widely used benchmark datasets, the method falls short for challenging scenarios like urban traffic. This work introduces temporal context into the state-of-the-art methods for sound source localization in urban scenes using optical flow as a means to encode motion information. An analysis of the strengths and weaknesses of our methods helps us better understand the problem of visual sound source localization and sheds light on open challenges for audio-visual scene understanding.