The Limits of Data Scaling: Sub-token Utilization and Acoustic Saturation in Multilingual ASR

TL;DR

This paper investigates how multilingual ASR models utilize sub-tokens across languages, revealing that sub-token discovery saturates quickly regardless of data disparity, and that linguistic features influence token usage more than training data volume.

Contribution

The study provides an empirical analysis of sub-token utilization in multilingual ASR, introducing the concept of acoustic saturation time and highlighting the influence of linguistic features on token discovery.

Findings

Sub-token discovery saturates exponentially across languages.

Data disparity does not significantly affect lexical diversity.

Linguistic and orthographic features influence token utilization patterns.

Abstract

How much audio is needed to fully observe a multilingual ASR model's learned sub-token inventory across languages, and does data disparity in multilingual pre-training affect how these tokens are utilized during inference? We address this question by analyzing Whisper's decoding behavior during inference across 49 languages. By logging decoding candidate sub-tokens and tracking their cumulative discovery over time, we study the utilization pattern of the model's sub-token space. Results show that the total number of discovered tokens remains largely independent of a language's pre-training hours, indicating that data disparity does not strongly influence lexical diversity in the model's hypothesis space. Sub-token discovery rates follow a consistent exponential saturation pattern across languages, suggesting a stable time window after which additional audio yields minimal new sub-token activation. We refer to this convergence threshold as acoustic saturation time (AST). Further analyses of rank-frequency distributions reveal Zipf-like patterns better modeled by a Zipf-Mandelbrot law, and mean sub-token length shows a positive correlation with resource level. Additionally, those metrics show more favorable patterns for languages in the Latin script than those in scripts such as Cyrillic, CJK, and Semitic. Together, our study suggests that sub-token utilization during multilingual ASR inference is constrained more by the statistical, typological, and orthographic structure of the speech than by training data scale, providing an empirical basis for more equitable corpus construction and cross-lingual evaluation.