Regularization Advantages of Multilingual Neural Language Models for Low Resource Domains

TL;DR

This paper introduces a multilingual neural language model that shares parameters across low-resource languages, acting as a regularizer and improving performance in data-scarce scenarios, especially for conversational data.

Contribution

The study proposes a novel multilingual neural language model architecture with shared and language-specific components, enhancing low-resource language modeling through transfer learning.

Findings

Significant performance improvements over monolingual models in low-resource settings

Effective cross-lingual transfer learning demonstrated across four languages

Multilingual model acts as a regularizer, benefiting very limited data scenarios

Abstract

Neural language modeling (LM) has led to significant improvements in several applications, including Automatic Speech Recognition. However, they typically require large amounts of training data, which is not available for many domains and languages. In this study, we propose a multilingual neural language model architecture, trained jointly on the domain-specific data of several low-resource languages. The proposed multilingual LM consists of language specific word embeddings in the encoder and decoder, and one language specific LSTM layer, plus two LSTM layers with shared parameters across the languages. This multilingual LM model facilitates transfer learning across the languages, acting as an extra regularizer in very low-resource scenarios. We integrate our proposed multilingual approach with a state-of-the-art highly-regularized neural LM, and evaluate on the conversational data domain for four languages over a range of training data sizes. Compared to monolingual LMs, the results show significant improvements of our proposed multilingual LM when the amount of available training data is limited, indicating the advantages of cross-lingual parameter sharing in very low-resource language modeling.