Syntax-augmented Multilingual BERT for Cross-lingual Transfer

TL;DR

This paper enhances multilingual BERT by explicitly incorporating syntactic dependency structures, significantly improving cross-lingual transfer performance across multiple NLP tasks and benchmarks.

Contribution

It introduces a syntax-augmented training method for mBERT that explicitly encodes syntactic dependencies, advancing cross-lingual transfer capabilities.

Findings

Improves cross-lingual transfer on benchmarks by 1.4-1.6 points.

Achieves larger gains of 3.1-3.9 points in generalized transfer settings.

Demonstrates effectiveness across tasks like classification, QA, NER, and parsing.

Abstract

In recent years, we have seen a colossal effort in pre-training multilingual text encoders using large-scale corpora in many languages to facilitate cross-lingual transfer learning. However, due to typological differences across languages, the cross-lingual transfer is challenging. Nevertheless, language syntax, e.g., syntactic dependencies, can bridge the typological gap. Previous works have shown that pre-trained multilingual encoders, such as mBERT \cite{devlin-etal-2019-bert}, capture language syntax, helping cross-lingual transfer. This work shows that explicitly providing language syntax and training mBERT using an auxiliary objective to encode the universal dependency tree structure helps cross-lingual transfer. We perform rigorous experiments on four NLP tasks, including text classification, question answering, named entity recognition, and task-oriented semantic parsing. The experiment results show that syntax-augmented mBERT improves cross-lingual transfer on popular benchmarks, such as PAWS-X and MLQA, by 1.4 and 1.6 points on average across all languages. In the \emph{generalized} transfer setting, the performance boosted significantly, with 3.9 and 3.1 points on average in PAWS-X and MLQA.