Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR

TL;DR

This paper explores how to efficiently incorporate conversational context into LLM-based speech recognition by compressing prior audio into learned latent tokens, improving recognition of contextual entities.

Contribution

It introduces Abstract Compression, a method that reduces audio context size while retaining key information, enhancing LLM-based ASR performance.

Findings

Compressed context recovers part of raw-context gains

Method improves recognition of contextual entities

Smaller prior-turn audio footprint with maintained performance

Abstract

Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.