FineLAP: Taming Heterogeneous Supervision for Fine-grained Language-Audio Pretraining

TL;DR

FineLAP introduces a novel training paradigm for audio-language models that enhances both clip- and frame-level understanding by leveraging heterogeneous supervision and a new dataset.

Contribution

It proposes a dual-stream sigmoid loss with cluster-based sampling and a large-scale synthetic dataset to improve fine-grained audio-language pretraining.

Findings

Achieves state-of-the-art results on multiple audio understanding tasks.

Demonstrates mutual benefits of coarse- and fine-grained alignment.

Introduces FineLAP-100k, a large synthetic SED dataset.

Abstract

Contrastively pretrained audio-language models (e.g., CLAP) excel at clip-level understanding but struggle with frame-level tasks. Existing extensions fail to exploit the varying granularity of real-world audio-text data, where massive clip-level textual descriptions coexist with limited frame-level annotations. This paper proposes Fine-grained Language-Audio Pretraining (FineLAP), a novel training paradigm that advances both clip- and frame-level alignment in CLAP with heterogeneous data. FineLAP introduces a dual-stream sigmoid loss with a cluster-based sampling strategy to jointly learn from clip- and frame-level supervision. To capture both global semantics and local details, FineLAP uses a decoupled audio projector on top of a self-supervised encoder. To alleviate the scarcity of temporally annotated data, we present FineLAP-100k, a large-scale synthetic SED dataset constructed through a scalable curation pipeline. Extensive experiments demonstrate that FineLAP achieves SOTA performance across multiple audio understanding tasks, including retrieval, classification, sound event detection, and text-to-audio grounding. Ablation studies further show that coarse- and fine-grained alignment are mutually beneficial, providing insights for building better audio-language models (ALMs).