Marco-ASR: A Principled and Metric-Driven Framework for Fine-Tuning Large-Scale ASR Models for Domain Adaptation

TL;DR

This paper introduces a metric-driven fine-tuning framework for large-scale ASR models, enhancing domain adaptation by optimizing learning rates and data strategies, validated across multiple models and datasets.

Contribution

It presents a novel, principled approach to fine-tuning large ASR models using performance metrics, addressing domain mismatch and overfitting issues.

Findings

Improved domain-specific ASR accuracy across multiple models.

Effective learning rate optimization enhances fine-tuning.

Framework prevents overfitting in diverse datasets.

Abstract

Automatic Speech Recognition (ASR) models have achieved remarkable accuracy in general settings, yet their performance often degrades in domain-specific applications due to data mismatch and linguistic variability. This challenge is amplified for modern Large Language Model (LLM)-based ASR systems, whose massive scale and complex training dynamics make effective fine-tuning non-trivial. To address this gap, this paper proposes a principled and metric-driven fine-tuning framework for adapting both traditional and LLM-based ASR models to specialized domains. The framework emphasizes learning rate optimization based on performance metrics, combined with domain-specific data transformation and augmentation. We empirically evaluate our framework on state-of-the-art models, including Whisper, Whisper-Turbo, and Qwen2-Audio, across multi-domain, multilingual, and multi-length datasets. Our results not only validate the proposed framework but also establish practical protocols for improving domain-specific ASR performance while preventing overfitting.