Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER

TL;DR

This paper enhances zero-resource cross-lingual NLP tasks by applying adversarial learning to multilingual BERT, improving performance and aligning embeddings across languages for classification and NER tasks.

Contribution

It introduces adversarial training to multilingual BERT, significantly improving zero-resource cross-lingual classification and NER performance and promoting embedding alignment across languages.

Findings

Performance improvements on MLDoc and CoNLL datasets

Adversarial training aligns English and translated document embeddings

Language-adversarial training boosts cross-lingual transfer

Abstract

Contextual word embeddings (e.g. GPT, BERT, ELMo, etc.) have demonstrated state-of-the-art performance on various NLP tasks. Recent work with the multilingual version of BERT has shown that the model performs very well in zero-shot and zero-resource cross-lingual settings, where only labeled English data is used to finetune the model. We improve upon multilingual BERT's zero-resource cross-lingual performance via adversarial learning. We report the magnitude of the improvement on the multilingual MLDoc text classification and CoNLL 2002/2003 named entity recognition tasks. Furthermore, we show that language-adversarial training encourages BERT to align the embeddings of English documents and their translations, which may be the cause of the observed performance gains.