From English to Code-Switching: Transfer Learning with Strong Morphological Clues

TL;DR

This paper introduces CS-ELMo, a transfer learning approach that adapts monolingual models to code-switched text, achieving state-of-the-art results in CS tasks like NER and POS tagging across multiple language pairs.

Contribution

The paper presents CS-ELMo, a novel extension of ELMo with a position-aware attention mechanism, effectively transferring English knowledge to code-switched language pairs.

Findings

Outperforms multilingual BERT and CS-unaware ELMo models

Establishes new state-of-the-art in CS NER and POS tagging

Effective transfer learning for multiple language pairs

Abstract

Linguistic Code-switching (CS) is still an understudied phenomenon in natural language processing. The NLP community has mostly focused on monolingual and multi-lingual scenarios, but little attention has been given to CS in particular. This is partly because of the lack of resources and annotated data, despite its increasing occurrence in social media platforms. In this paper, we aim at adapting monolingual models to code-switched text in various tasks. Specifically, we transfer English knowledge from a pre-trained ELMo model to different code-switched language pairs (i.e., Nepali-English, Spanish-English, and Hindi-English) using the task of language identification. Our method, CS-ELMo, is an extension of ELMo with a simple yet effective position-aware attention mechanism inside its character convolutions. We show the effectiveness of this transfer learning step by outperforming multilingual BERT and homologous CS-unaware ELMo models and establishing a new state of the art in CS tasks, such as NER and POS tagging. Our technique can be expanded to more English-paired code-switched languages, providing more resources to the CS community.