Sound Event Detection Guided by Semantic Contexts of Scenes

TL;DR

This paper introduces a scene-informed sound event detection method that leverages large-scale language models to utilize unseen semantic scene contexts, improving detection accuracy on standard datasets.

Contribution

It proposes a novel scene-agnostic approach using pre-trained language models for more flexible and accurate sound event detection across diverse scene contexts.

Findings

Improved micro and macro F-scores by over 4 percentage points.

Utilized large-scale language models for semantic scene representation.

Validated on TUT datasets with significant performance gains.

Abstract

Some studies have revealed that contexts of scenes (e.g., "home," "office," and "cooking") are advantageous for sound event detection (SED). Mobile devices and sensing technologies give useful information on scenes for SED without the use of acoustic signals. However, conventional methods can employ pre-defined contexts in inference stages but not undefined contexts. This is because one-hot representations of pre-defined scenes are exploited as prior contexts for such conventional methods. To alleviate this problem, we propose scene-informed SED where pre-defined scene-agnostic contexts are available for more accurate SED. In the proposed method, pre-trained large-scale language models are utilized, which enables SED models to employ unseen semantic contexts of scenes in inference stages. Moreover, we investigated the extent to which the semantic representation of scene contexts is useful for SED. Experimental results performed with TUT Sound Events 2016/2017 and TUT Acoustic Scenes 2016/2017 datasets show that the proposed method improves micro and macro F-scores by 4.34 and 3.13 percentage points compared with conventional Conformer- and CNN--BiGRU-based SED, respectively.