Rethinking Entropy Allocation in LLM-based ASR: Understanding the Dynamics between Speech Encoders and LLMs

TL;DR

This paper analyzes entropy allocation in LLM-based ASR, proposing a multi-stage training strategy to improve efficiency and reduce hallucinations, achieving competitive results with fewer parameters.

Contribution

It introduces three entropy allocation metrics and a novel training approach that enhances parameter efficiency and hallucination robustness in LLM-based ASR.

Findings

Achieves state-of-the-art performance with only 2.3B parameters.

Effectively mitigates hallucinations through decoupling-oriented design.

Redesigns pretraining to address speech-text modality gap.

Abstract

Integrating large language models (LLMs) into automatic speech recognition (ASR) has become a dominant paradigm. Although recent LLM-based ASR models have shown promising performance on public benchmarks, it remains challenging to balance recognition quality with latency and overhead, while hallucinations further limit real-world deployment. In this study, we revisit LLM-based ASR from an entropy allocation perspective and introduce three metrics to characterize how training paradigms allocate entropy reduction between the speech encoder and the LLM. To remedy entropy-allocation inefficiencies in prevailing approaches, we propose a principled multi-stage training strategy grounded in capability-boundary awareness, optimizing parameter efficiency and hallucination robustness. Specifically, we redesign the pretraining strategy to alleviate the speech-text modality gap, and further introduce an iterative asynchronous SFT stage between alignment and joint SFT to preserve functional decoupling and constrain encoder representation drift. Experiments on Mandarin and English benchmarks show that our method achieves competitive performance with state-of-the-art models using only 2.3B parameters, while also effectively mitigating hallucinations through our decoupling-oriented design.