Speech-XL: Towards Long-Form Speech Understanding in Large Speech Language Models

TL;DR

Speech-XL introduces a novel approach to long-form speech understanding by leveraging key-value sparsification and a special summarization token, enabling efficient compression and processing of extended audio sequences in large speech language models.

Contribution

The paper proposes Speech-XL, which uses a new Speech Summarization Token and curriculum learning to effectively compress long speech inputs for large models, addressing key bottlenecks.

Findings

Achieves competitive performance on LongSpeech and AUDIOMARATHON benchmarks.

Utilizes significantly less training data than baseline models.

Effectively compresses long-form speech with high-ratio summarization.

Abstract

Despite the growing success of Large Speech Language Models (LSLMs) in processing short-term acoustic signals, their extension to long-form audio understanding is severely bottlenecked. This limitation stems from the limited context length and the exorbitant memory footprints required for long-form inference. In this work, we propose Speech-XL, a new model that capitalizes on the intrinsic key-value (KV) sparsification capacity of Large Language Models (LLMs) to achieve high-ratio speech input compression. Specifically, we introduce a novel special token, the Speech Summarization Token (SST), for each speech interval to encapsulate the intra-interval speech information into its associated KV pairs. The SST module is trained via instruction fine-tuning, employing a curriculum learning strategy where the SST learns to compress information in a progressive manner--advancing from low-ratio (simple) to high-ratio (challenging) compression. Despite utilizing significantly less training data than other baselines, our model achieves highly competitive performance on major benchmarks, including LongSpeech and AUDIOMARATHON. By addressing the long-standing bottlenecks in long-form audio modeling, our approach offers a novel perspective on the condensation of extensive acoustic sequences.