Multilingual Text Representation

TL;DR

This survey reviews the evolution of multilingual text representation models, highlighting their capabilities, challenges, and potential for democratizing language technology across diverse languages and dialects.

Contribution

It provides a comprehensive overview of the development, current state, and future directions of multilingual text representation models in NLP.

Findings

Multilingual models now perform across over 100 languages.

Progress from simple word representations to complex language understanding.

Challenges remain in achieving equitable and inclusive language coverage.

Abstract

Modern NLP breakthrough includes large multilingual models capable of performing tasks across more than 100 languages. State-of-the-art language models came a long way, starting from the simple one-hot representation of words capable of performing tasks like natural language understanding, common-sense reasoning, or question-answering, thus capturing both the syntax and semantics of texts. At the same time, language models are expanding beyond our known language boundary, even competitively performing over very low-resource dialects of endangered languages. However, there are still problems to solve to ensure an equitable representation of texts through a unified modeling space across language and speakers. In this survey, we shed light on this iterative progression of multilingual text representation and discuss the driving factors that ultimately led to the current state-of-the-art. Subsequently, we discuss how the full potential of language democratization could be obtained, reaching beyond the known limits and what is the scope of improvement in that space.