Semantic Proximity Alignment: Towards Human Perception-consistent Audio Tagging by Aligning with Label Text Description

TL;DR

This paper introduces Semantic Proximity Alignment (SPA), a method that enhances audio tagging models by aligning audio features with label text descriptions, incorporating semantic hierarchy and proximity to improve performance and human perception consistency.

Contribution

The paper proposes SPA, a novel training approach that uses auxiliary text descriptions to embed semantic relationships into audio tagging models, outperforming traditional one-hot label training.

Findings

SPA improves OmAP by 1.8 points over baseline.

Models trained with SPA show higher human perception alignment.

Semantic proximity information enhances audio tagging accuracy.

Abstract

Most audio tagging models are trained with one-hot labels as supervised information. However, one-hot labels treat all sound events equally, ignoring the semantic hierarchy and proximity relationships between sound events. In contrast, the event descriptions contains richer information, describing the distance between different sound events with semantic proximity. In this paper, we explore the impact of training audio tagging models with auxiliary text descriptions of sound events. By aligning the audio features with the text features of corresponding labels, we inject the hierarchy and proximity information of sound events into audio encoders, improving the performance while making the prediction more consistent with human perception. We refer to this approach as Semantic Proximity Alignment (SPA). We use Ontology-aware mean Average Precision (OmAP) as the main evaluation metric for the models. OmAP reweights the false positives based on Audioset ontology distance and is more consistent with human perception compared to mAP. Experimental results show that the audio tagging models trained with SPA achieve higher OmAP compared to models trained with one-hot labels solely (+1.8 OmAP). Human evaluations also demonstrate that the predictions of SPA models are more consistent with human perception.