Call Larisa Ivanovna: Code-Switching Fools Multilingual NLU Models

TL;DR

This paper highlights the lack of code-switched data in multilingual NLU benchmarks, introduces a synthetic code-switched test set, and demonstrates that current models struggle with code-switching, but pre-training on synthetic data can improve performance.

Contribution

The authors create a synthetic code-switched dataset for NLU evaluation and analyze its impact on model performance, revealing challenges and potential improvements for handling code-switching.

Findings

State-of-the-art NLU models' performance drops significantly on code-switched data.

Pre-training on synthetic code-mixed data helps maintain NLU performance.

Closer languages result in better handling of code-switching by models.

Abstract

Practical needs of developing task-oriented dialogue assistants require the ability to understand many languages. Novel benchmarks for multilingual natural language understanding (NLU) include monolingual sentences in several languages, annotated with intents and slots. In such setup models for cross-lingual transfer show remarkable performance in joint intent recognition and slot filling. However, existing benchmarks lack of code-switched utterances, which are difficult to gather and label due to complexity in the grammatical structure. The evaluation of NLU models seems biased and limited, since code-switching is being left out of scope. Our work adopts recognized methods to generate plausible and naturally-sounding code-switched utterances and uses them to create a synthetic code-switched test set. Based on experiments, we report that the state-of-the-art NLU models are unable to handle code-switching. At worst, the performance, evaluated by semantic accuracy, drops as low as 15\% from 80\% across languages. Further we show, that pre-training on synthetic code-mixed data helps to maintain performance on the proposed test set at a comparable level with monolingual data. Finally, we analyze different language pairs and show that the closer the languages are, the better the NLU model handles their alternation. This is in line with the common understanding of how multilingual models conduct transferring between languages