ILT-Iterative LoRA Training through Focus-Feedback-Fix for Multilingual Speech Recognition

TL;DR

This paper introduces ILT, an iterative training method with focus-feedback-fix stages, to improve multilingual speech recognition by addressing overfitting in LoRA fine-tuning, demonstrating strong experimental results and competitive challenge performance.

Contribution

The paper presents a novel ILT training paradigm combined with pseudo labeling to enhance LoRA fine-tuning for multilingual speech recognition, improving performance bounds.

Findings

Effective in reducing overfitting in LoRA fine-tuning.

Achieved top rankings in Interspeech 2025 challenge.

Demonstrated practical applicability in multilingual ASR.

Abstract

The deep integration of large language models and automatic speech recognition systems has become a promising research direction with high practical value. To address the overfitting issue commonly observed in Low-Rank Adaptation (LoRA) during the supervised fine-tuning (SFT) stage, this work proposes an innovative training paradigm Iterative LoRA Training (ILT) in combination with an Iterative Pseudo Labeling strategy, effectively enhancing the theoretical upper bound of model performance. Based on Whisper-large-v3 and Qwen2-Audio, we conduct systematic experiments using a three-stage training process: Focus Training, Feed Back Training, and Fix Training. Experimental results demonstrate the effectiveness of the proposed method. Furthermore, the MegaAIS research team applied this technique in the Interspeech 2025 Multilingual Conversational Speech Language Modeling Challenge (MLC-SLM), achieving 4th in Track 1 (Multilingual ASR Task) and 1st place in Track 2 (Speech Separation and Recognition Task), showcasing the practical feasibility and strong application potential of our approach.