Incorporating Lexical and Syntactic Knowledge for Unsupervised Cross-Lingual Transfer

TL;DR

This paper introduces a novel framework that combines lexical and syntactic knowledge to improve unsupervised cross-lingual transfer, demonstrating consistent performance gains across multiple NLP tasks.

Contribution

It presents a new approach integrating lexical and syntactic information using code-switching and graph attention networks within multilingual BERT for better transfer.

Findings

Outperforms all baseline models in zero-shot transfer tasks

Achieves 1.0 to 3.7 points improvement on text classification, NER, and semantic parsing

Demonstrates the effectiveness of combining lexical and syntactic knowledge

Abstract

Unsupervised cross-lingual transfer involves transferring knowledge between languages without explicit supervision. Although numerous studies have been conducted to improve performance in such tasks by focusing on cross-lingual knowledge, particularly lexical and syntactic knowledge, current approaches are limited as they only incorporate syntactic or lexical information. Since each type of information offers unique advantages and no previous attempts have combined both, we attempt to explore the potential of this approach. In this paper, we present a novel framework called "Lexicon-Syntax Enhanced Multilingual BERT" that combines both lexical and syntactic knowledge. Specifically, we use Multilingual BERT (mBERT) as the base model and employ two techniques to enhance its learning capabilities. The code-switching technique is used to implicitly teach the model lexical alignment information, while a syntactic-based graph attention network is designed to help the model encode syntactic structure. To integrate both types of knowledge, we input code-switched sequences into both the syntactic module and the mBERT base model simultaneously. Our extensive experimental results demonstrate this framework can consistently outperform all baselines of zero-shot cross-lingual transfer, with the gains of 1.0~3.7 points on text classification, named entity recognition (ner), and semantic parsing tasks. Keywords:cross-lingual transfer, lexicon, syntax, code-switching, graph attention network