FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion

TL;DR

This paper introduces a novel two-stage pipeline for fine-grained, context-aware audio captioning that leverages multimodal cues and large language models, supported by a new large-scale dataset called FusionAudio-1.2M.

Contribution

It presents a scalable method for detailed audio captioning, introduces the FusionAudio dataset with 1.2 million captions, and develops improved audio models with better audio-text alignment.

Findings

Enhanced audio captioning accuracy with multimodal fusion

FusionAudio dataset enables better training of audio models

Improved audio-text alignment using CLAP-based encoder

Abstract

High-quality, large-scale audio captioning is crucial for advancing audio understanding, yet current automated methods often generate captions that lack fine-grained detail and contextual accuracy, primarily due to their reliance on limited unimodal or superficial multimodal information. Drawing inspiration from human auditory perception, which adeptly integrates cross-modal cues and performs sophisticated auditory scene analysis, we introduce a novel two-stage automated pipeline. This pipeline first employs specialized pretrained models to extract diverse contextual cues (e.g., speech, music, general sounds, and visual information from associated video). A large language model (LLM) then synthesizes these rich, multimodal inputs to generate detailed and context-aware audio captions. Key contributions of this work include: (1) the proposed scalable method for fine-grained audio caption generation; (2) FusionAudio, a new large-scale dataset comprising 1.2 million such detailed captions, combined with 6 million QA pairs; and (3) enhanced audio models developed using FusionAudio, specifically a CLAP-based audio encoder with superior audio-text alignment and instruction following. This paper paves the way for more nuanced and accurate automated understanding of complex audio environments. Code and data can be found in https://github.com/satsuki2486441738/FusionAudio.