Discrete Tokens Exhibit Interlanguage Speech Intelligibility Benefit: an Analytical Study Towards Accent-robust ASR Only with Native Speech Data

TL;DR

This paper demonstrates that discrete token-based ASR models trained solely on native speech data exhibit the interlanguage speech intelligibility benefit, improving recognition of non-native speech without requiring non-native data.

Contribution

It provides an analytical study showing that discrete tokens from SSL models can achieve accent-robust ASR by leveraging the ISIB phenomenon with only native speech data.

Findings

ISIB occurs in discrete token-based ASR

Native data training improves non-native speech recognition

Approach applicable to various accents with scarce data

Abstract

In this study, we gained insight that contributes to achieving accent-robust ASR using only native speech data. In human perception of non-native speech, the phenomenon known as "interlanguage speech intelligibility benefit" (ISIB) is observed, where non-native listeners who share the native language with the speaker understand the speech better compared even to native listeners. Based on the idea that discrete tokens extracted from self-supervised learning (SSL) models represent the human perception of speech, we conducted an analytical study on the robustness of discrete token-based ASR to non-native speech, varying the language used for training the tokenization, which is viewed as a technical implementation of ISIB. The results showed that ISIB actually occurred in the discrete token-based ASR. Since our approach relies only on native speech data to simulate the behavior of human perception, it is expected to be applicable to a wide range of accents for which speech data is scarce.