BhashaSetu: Cross-Lingual Knowledge Transfer from High-Resource to Extreme Low-Resource Languages

TL;DR

This paper introduces a novel graph-based method, GETR, for cross-lingual knowledge transfer that significantly improves NLP task performance in extremely low-resource languages by leveraging high-resource language data.

Contribution

The paper proposes GETR, a graph-enhanced token representation method, and demonstrates its effectiveness over existing baselines in low-resource language NLP tasks.

Findings

GETR outperforms baselines by 13-27 percentage points in key NLP tasks.

Significant improvements in POS tagging, sentiment analysis, and NER for low-resource languages.

Analysis reveals key transfer mechanisms and factors for successful knowledge transfer.

Abstract

Despite remarkable advances in natural language processing, developing effective systems for low-resource languages remains a formidable challenge, with performances typically lagging far behind high-resource counterparts due to data scarcity and insufficient linguistic resources. Cross-lingual knowledge transfer has emerged as a promising approach to address this challenge by leveraging resources from high-resource languages. In this paper, we investigate methods for transferring linguistic knowledge from high-resource languages to low-resource languages, where the number of labeled training instances is in hundreds. We focus on sentence-level and word-level tasks. We introduce a novel method, GETR (Graph-Enhanced Token Representation) for cross-lingual knowledge transfer along with two adopted baselines (a) augmentation in hidden layers and (b) token embedding transfer through token translation. Experimental results demonstrate that our GNN-based approach significantly outperforms existing multilingual and cross-lingual baseline methods, achieving 13 percentage point improvements on truly low-resource languages (Mizo, Khasi) for POS tagging, and 20 and 27 percentage point improvements in macro-F1 on simulated low-resource languages (Marathi, Bangla, Malayalam) across sentiment classification and NER tasks respectively. We also present a detailed analysis of the transfer mechanisms and identify key factors that contribute to successful knowledge transfer in this linguistic context.