Efficient Scaling for LLM-based ASR

TL;DR

This paper introduces EFIN, a multi-stage training strategy for LLM-based ASR that improves efficiency and performance by pretraining the speech encoder before LLM integration, supported by a derived scaling law.

Contribution

The paper proposes EFIN, a novel multi-stage training method for LLM-ASR, and derives a scaling law to guide efficient model scaling.

Findings

EFIN outperforms joint post-training with 21.1% CERR reduction.

EFIN reduces computational costs by nearly 50% FLOPs.

A scaling law approximates ASR error as a function of computation.

Abstract

Large language model (LLM)-based automatic speech recognition (ASR) achieves strong performance but often incurs high computational costs. This work investigates how to obtain the best LLM-ASR performance efficiently. Through comprehensive and controlled experiments, we find that pretraining the speech encoder before integrating it with the LLM leads to significantly better scaling efficiency than the standard practice of joint post-training of LLM-ASR. Based on this insight, we propose a new multi-stage LLM-ASR training strategy, EFIN: Encoder First Integration. Among all training strategies evaluated, EFIN consistently delivers better performance (relative to 21.1% CERR) with significantly lower computation budgets (49.9% FLOPs). Furthermore, we derive a scaling law that approximates ASR error rates as a computation function, providing practical guidance for LLM-ASR scaling.