Code-switched Language Models Using Dual RNNs and Same-Source Pretraining

TL;DR

This paper introduces dual RNN-based language models with same-source pretraining to improve code-switched language modeling, demonstrating significant perplexity reductions on Mandarin-English data.

Contribution

It presents a novel dual RNN architecture and a pretraining method using synthetic data, advancing code-switched language modeling techniques.

Findings

Perplexity reduced significantly on Mandarin-English task

Dual RNN units effectively model each language in code-switching

Pretraining with synthetic data improves language model performance

Abstract

This work focuses on building language models (LMs) for code-switched text. We propose two techniques that significantly improve these LMs: 1) A novel recurrent neural network unit with dual components that focus on each language in the code-switched text separately 2) Pretraining the LM using synthetic text from a generative model estimated using the training data. We demonstrate the effectiveness of our proposed techniques by reporting perplexities on a Mandarin-English task and derive significant reductions in perplexity.