AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration

TL;DR

AuditoryHuM introduces an unsupervised, human-MLLM collaborative framework for generating and clustering auditory scene labels, reducing manual effort and enabling scalable, edge-deployable scene recognition models.

Contribution

The paper presents a novel collaborative approach combining MLLMs and human input for automatic auditory scene label discovery and clustering, improving scalability and label quality.

Findings

Effective label generation across diverse datasets

Improved clustering cohesion with thematic balance

Facilitates training of lightweight scene recognition models

Abstract

Manual annotation of audio datasets is labour intensive, and it is challenging to balance label granularity with acoustic separability. We introduce AuditoryHuM, a novel framework for the unsupervised discovery and clustering of auditory scene labels using a collaborative Human-Multimodal Large Language Model (MLLM) approach. By leveraging MLLMs (Gemma and Qwen) the framework generates contextually relevant labels for audio data. To ensure label quality and mitigate hallucinations, we employ zero-shot learning techniques (Human-CLAP) to quantify the alignment between generated text labels and raw audio content. A strategically targeted human-in-the-loop intervention is then used to refine the least aligned pairs. The discovered labels are grouped into thematically cohesive clusters using an adjusted silhouette score that incorporates a penalty parameter to balance cluster cohesion and thematic granularity. Evaluated across three diverse auditory scene datasets (ADVANCE, AHEAD-DS, and TAU 2019), AuditoryHuM provides a scalable, low-cost solution for creating standardised taxonomies. This solution facilitates the training of lightweight scene recognition models deployable to edge devices, such as hearing aids and smart home assistants. The project page and code: https://github.com/Australian-Future-Hearing-Initiative