LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech

TL;DR

LongSpeech is a comprehensive benchmark designed to evaluate speech models on long-duration audio tasks, highlighting current limitations and guiding future research in long-form speech understanding.

Contribution

We introduce LongSpeech, a large-scale, scalable benchmark with diverse annotations for multiple long-form speech tasks, and a pipeline for future extensions.

Findings

State-of-the-art models perform poorly on long speech tasks.

Models tend to specialize in individual tasks rather than generalize.

High-level reasoning remains challenging for current models.

Abstract

Recent advances in audio-language models have demonstrated remarkable success on short, segment-level speech tasks. However, real-world applications such as meeting transcription, spoken document understanding, and conversational analysis require robust models capable of processing and reasoning over long-form audio. In this work, we present LongSpeech, a large-scale and scalable benchmark specifically designed to evaluate and advance the capabilities of speech models on long-duration audio. LongSpeech comprises over 100,000 speech segments, each approximately 10 minutes long, with rich annotations for ASR, speech translation, summarization, language detection, speaker counting, content separation, and question answering. We introduce a reproducible pipeline for constructing long-form speech benchmarks from diverse sources, enabling future extensions. Our initial experiments with state-of-the-art models reveal significant performance gaps, with models often specializing in one task at the expense of others and struggling with higher-level reasoning. These findings underscore the challenging nature of our benchmark. Our benchmark will be made publicly available to the research community.