The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding

TL;DR

This paper demonstrates that bilingual intermediate pretraining significantly improves performance on multiple code-switched NLP tasks across various language pairs, outperforming previous methods and standard pretraining techniques.

Contribution

It introduces bilingual intermediate pretraining as an effective method to enhance code-switched NLP task performance, with consistent gains across multiple tasks and language pairs.

Findings

Achieved up to 20.15% improvement in F1 scores for code-switched NLP tasks.

Demonstrated consistent performance gains across four language pairs.

Proposed a code-switched MLM pretraining technique that outperforms standard MLM.

Abstract

While recent benchmarks have spurred a lot of new work on improving the generalization of pretrained multilingual language models on multilingual tasks, techniques to improve code-switched natural language understanding tasks have been far less explored. In this work, we propose the use of bilingual intermediate pretraining as a reliable technique to derive large and consistent performance gains on three different NLP tasks using code-switched text. We achieve substantial absolute improvements of 7.87%, 20.15%, and 10.99%, on the mean accuracies and F1 scores over previous state-of-the-art systems for Hindi-English Natural Language Inference (NLI), Question Answering (QA) tasks, and Spanish-English Sentiment Analysis (SA) respectively. We show consistent performance gains on four different code-switched language-pairs (Hindi-English, Spanish-English, Tamil-English and Malayalam-English) for SA. We also present a code-switched masked language modelling (MLM) pretraining technique that consistently benefits SA compared to standard MLM pretraining using real code-switched text.