SUTA-LM: Bridging Test-Time Adaptation and Language Model Rescoring for Robust ASR

TL;DR

This paper introduces SUTA-LM, a method that effectively combines test-time adaptation with language model rescoring to improve the robustness of automatic speech recognition across diverse domains.

Contribution

It presents SUTA-LM, a novel extension of entropy-minimization-based TTA that incorporates language model rescoring with an auto-step selection mechanism for better domain adaptation.

Findings

SUTA-LM outperforms baseline methods on 18 diverse ASR datasets.

The combined approach improves robustness against domain mismatches.

Auto-step selection enhances the effectiveness of test-time adaptation.

Abstract

Despite progress in end-to-end ASR, real-world domain mismatches still cause performance drops, which Test-Time Adaptation (TTA) aims to mitigate by adjusting models during inference. Recent work explores combining TTA with external language models, using techniques like beam search rescoring or generative error correction. In this work, we identify a previously overlooked challenge: TTA can interfere with language model rescoring, revealing the nontrivial nature of effectively combining the two methods. Based on this insight, we propose SUTA-LM, a simple yet effective extension of SUTA, an entropy-minimization-based TTA approach, with language model rescoring. SUTA-LM first applies a controlled adaptation process guided by an auto-step selection mechanism leveraging both acoustic and linguistic information, followed by language model rescoring to refine the outputs. Experiments on 18 diverse ASR datasets show that SUTA-LM achieves robust results across a wide range of domains.