Audiocards: Structured Metadata Improves Audio Language Models For Sound Design

TL;DR

Audiocards introduces structured metadata grounded in acoustic attributes, leveraging large language models to enhance sound search, captioning, and metadata generation in sound design applications.

Contribution

The paper presents audiocards, a novel structured metadata format for sound effects, improving audio retrieval and captioning by utilizing LLMs trained on this metadata.

Findings

Improved text-audio retrieval performance

Enhanced sound effect captioning accuracy

Better metadata generation for sound libraries

Abstract

Sound designers search for sounds in large sound effects libraries using aspects such as sound class or visual context. However, the metadata needed for such search is often missing or incomplete, and requires significant manual effort to add. Existing solutions to automate this task by generating metadata, i.e. captioning, and search using learned embeddings, i.e. text-audio retrieval, are not trained on metadata with the structure and information pertinent to sound design. To this end we propose audiocards, structured metadata grounded in acoustic attributes and sonic descriptors, by exploiting the world knowledge of LLMs. We show that training on audiocards improves downstream text-audio retrieval, descriptive captioning, and metadata generation on professional sound effects libraries. Moreover, audiocards also improve performance on general audio captioning and retrieval over the baseline single-sentence captioning approach. We release a curated dataset of sound effects audiocards to invite further research in audio language modeling for sound design.