Selective Invocation for Multilingual ASR: A Cost-effective Approach Adapting to Speech Recognition Difficulty

TL;DR

This paper introduces SIMA, a cost-effective selective invocation method for multilingual ASR that dynamically decides whether to transcribe speech directly or invoke a high-quality model, reducing costs and improving accuracy.

Contribution

The paper presents a novel selective invocation approach based on a spoken large language model, improving multilingual ASR efficiency and accuracy over traditional language identification methods.

Findings

Reduces word error rate by 18.7% compared to SLLM

Halves invocation costs relative to LID-based methods

Effective across multiple datasets

Abstract

Although multilingual automatic speech recognition (ASR) systems have significantly advanced, enabling a single model to handle multiple languages, inherent linguistic differences and data imbalances challenge SOTA performance across all languages. While language identification (LID) models can route speech to the appropriate ASR model, they incur high costs from invoking SOTA commercial models and suffer from inaccuracies due to misclassification. To overcome these, we propose SIMA, a selective invocation for multilingual ASR that adapts to the difficulty level of the input speech. Built on a spoken large language model (SLLM), SIMA evaluates whether the input is simple enough for direct transcription or requires the invocation of a SOTA ASR model. Our approach reduces word error rates by 18.7% compared to the SLLM and halves invocation costs compared to LID-based methods. Tests on three datasets show that SIMA is a scalable, cost-effective solution for multilingual ASR applications.