Exploring Differences between Human Perception and Model Inference in Audio Event Recognition

TL;DR

This paper investigates the discrepancies between human auditory perception and model inference in audio event recognition, introducing a new dataset and analysis to understand how models differ from human perception in identifying and detecting audio events.

Contribution

The paper presents the MAFAR dataset with multi-annotator labels, and analyzes the differences between human perception and model inference in semantic importance and event detection.

Findings

Humans ignore subtle or trivial events in semantic identification.

Models are affected by noisy events and tend to be more sensitive in event detection.

Significant gap exists between human perception and model inference in AER.

Abstract

Audio Event Recognition (AER) traditionally focuses on detecting and identifying audio events. Most existing AER models tend to detect all potential events without considering their varying significance across different contexts. This makes the AER results detected by existing models often have a large discrepancy with human auditory perception. Although this is a critical and significant issue, it has not been extensively studied by the Detection and Classification of Sound Scenes and Events (DCASE) community because solving it is time-consuming and labour-intensive. To address this issue, this paper introduces the concept of semantic importance in AER, focusing on exploring the differences between human perception and model inference. This paper constructs a Multi-Annotated Foreground Audio Event Recognition (MAFAR) dataset, which comprises audio recordings labelled by 10 professional annotators. Through labelling frequency and variance, the MAFAR dataset facilitates the quantification of semantic importance and analysis of human perception. By comparing human annotations with the predictions of ensemble pre-trained models, this paper uncovers a significant gap between human perception and model inference in both semantic identification and existence detection of audio events. Experimental results reveal that human perception tends to ignore subtle or trivial events in the event semantic identification, while model inference is easily affected by events with noises. Meanwhile, in event existence detection, models are usually more sensitive than humans.