# Polyglot Contextual Representations Improve Crosslingual Transfer

## TL;DR

This paper presents Rosita, a multilingual contextual language model trained on multiple languages, which improves cross-lingual transfer in tasks like parsing and named entity recognition by sharing representations across languages.

## Contribution

The paper introduces Rosita, a novel multilingual contextual language model trained on multiple languages, demonstrating its effectiveness in cross-lingual transfer tasks.

## Key findings

- Rosita outperforms monolingual models in cross-lingual tasks.
- Multilingual training enhances performance in dependency parsing and NER.
- Shared representations benefit low-resource languages.

## Abstract

We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages. Our method combines the advantages of contextual word representations with those of multilingual representation learning. We produce language models from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, semantic role labeling, and named entity recognition, with comparisons to monolingual and non-contextual variants. Our results provide further evidence for the benefits of polyglot learning, in which representations are shared across multiple languages.