Towards Fine-Grained and Multi-Granular Contrastive Language-Speech Pre-training

TL;DR

This paper introduces FCaps, a large-scale dataset with fine-grained speech annotations, and CLSP, a contrastive pre-training model that learns multi-granular speech-text representations for various tasks.

Contribution

The paper presents a novel dataset with detailed style annotations and a contrastive pre-training model that captures fine-grained and multi-granular speech-text relationships.

Findings

CLSP achieves reliable performance in speech-text retrieval and classification.

Annotations surpass existing datasets in correctness, coverage, and naturalness.

Model aligns well with human judgments across multiple tasks.

Abstract

Modeling fine-grained speaking styles remains challenging for language-speech representation pre-training, as existing speech-text models are typically trained with coarse captions or task-specific supervision, and scalable fine-grained style annotations are unavailable. We present FCaps, a large-scale dataset with fine-grained free-text style descriptions, encompassing 47k hours of speech and 19M fine-grained captions annotated via a novel end-to-end pipeline that directly grounds detailed captions in audio, thereby avoiding the error propagation caused by LLM-based rewriting in existing cascaded pipelines. Evaluations using LLM-as-a-judge demonstrate that our annotations surpass existing cascaded annotations in terms of correctness, coverage, and naturalness. Building on FCaps, we propose CLSP, a contrastive language-speech pre-trained model that integrates global and fine-grained supervision, enabling unified representations across multiple granularities. Extensive experiments demonstrate that CLSP learns fine-grained and multi-granular speech-text representations that perform reliably across global and fine-grained speech-text retrieval, zero-shot paralinguistic classification, and speech style similarity scoring, with strong alignment to human judgments. Code and dataset are publicly available at https://github.com/yfyeung/CLSP.