PIER: A Novel Metric for Evaluating What Matters in Code-Switching

TL;DR

This paper introduces PIER, a new metric for evaluating code-switching speech recognition that better captures performance on words of interest, addressing limitations of traditional metrics like WER.

Contribution

The paper proposes PIER, a novel metric focusing on words of interest in code-switching, providing a more accurate assessment of model performance.

Findings

Fine-tuning improves classical metrics but worsens code-switched word recognition.

PIER more accurately reflects code-switching performance.

Significant room for improvement in current models' handling of code-switching.

Abstract

Code-switching, the alternation of languages within a single discourse, presents a significant challenge for Automatic Speech Recognition. Despite the unique nature of the task, performance is commonly measured with established metrics such as Word-Error-Rate (WER). However, in this paper, we question whether these general metrics accurately assess performance on code-switching. Specifically, using both Connectionist-Temporal-Classification and Encoder-Decoder models, we show fine-tuning on non-code-switched data from both matrix and embedded language improves classical metrics on code-switching test sets, although actual code-switched words worsen (as expected). Therefore, we propose Point-of-Interest Error Rate (PIER), a variant of WER that focuses only on specific words of interest. We instantiate PIER on code-switched utterances and show that this more accurately describes the code-switching performance, showing huge room for improvement in future work. This focused evaluation allows for a more precise assessment of model performance, particularly in challenging aspects such as inter-word and intra-word code-switching.