Code Switching Language Model Using Monolingual Training Data

TL;DR

This paper presents a method to train a code-switching language model solely with monolingual data by alternating batches of different languages, significantly reducing perplexity and improving performance.

Contribution

The study introduces a novel approach of using alternate monolingual batches and MSE optimization to effectively train a code-switching language model without code-switching data.

Findings

Perplexity reduced from 299.63 to 80.38 using proposed methods.

Alternating monolingual batches improves code-switching language model performance.

Combined approach achieves results comparable to models trained on code-switch data.

Abstract

Training a code-switching (CS) language model using only monolingual data is still an ongoing research problem. In this paper, a CS language model is trained using only monolingual training data. As recurrent neural network (RNN) models are best suited for predicting sequential data. In this work, an RNN language model is trained using alternate batches from only monolingual English and Spanish data and the perplexity of the language model is computed. From the results, it is concluded that using alternate batches of monolingual data in training reduced the perplexity of a CS language model. The results were consistently improved using mean square error (MSE) in the output embeddings of RNN based language model. By combining both methods, perplexity is reduced from 299.63 to 80.38. The proposed methods were comparable to the language model fine tune with code-switch training data.