SALSA: Speedy ASR-LLM Synchronous Aggregation

TL;DR

SALSA is a novel method that efficiently couples ASR and LLM decoders for improved low-resource language recognition, achieving significant WER reductions without extensive retraining.

Contribution

It introduces a simple, training-efficient coupling technique for ASR and LLM decoders using projection and cascading tokenization.

Findings

Up to 38% WER reduction on low-resource languages

Efficient coupling with simple projection method

Effective handling of tokenizer mismatch

Abstract

Harnessing pre-trained LLMs to improve ASR systems, particularly for low-resource languages, is now an emerging area of research. Existing methods range from using LLMs for ASR error correction to tightly coupled systems that replace the ASR decoder with the LLM. These approaches either increase decoding time or require expensive training of the cross-attention layers. We propose SALSA, which couples the decoder layers of the ASR to the LLM decoder, while synchronously advancing both decoders. Such coupling is performed with a simple projection of the last decoder state, and is thus significantly more training efficient than earlier approaches. A challenge of our proposed coupling is handling the mismatch between the tokenizers of the LLM and ASR systems. We handle this mismatch using cascading tokenization with respect to the LLM and ASR vocabularies. We evaluate SALSA on 8 low-resource languages in the FLEURS benchmark, yielding substantial WER reductions of up to 38%.