Cross-Lingual NER for Financial Transaction Data in Low-Resource Languages

TL;DR

This paper introduces an efficient cross-lingual NER framework that leverages knowledge distillation and consistency training to transfer entity recognition capabilities from English to low-resource languages like Arabic using minimal labeled data.

Contribution

It presents a novel combination of knowledge distillation and unsupervised consistency training for cross-lingual NER in semi-structured data, effective with very few labeled samples.

Findings

Model performs well with only 30 labeled samples in the target language.

Outperforms state-of-the-art approaches like DistilBERT and supervised models trained on target language.

Enables entity recognition transfer from English to Arabic in semi-structured banking data.

Abstract

We propose an efficient modeling framework for cross-lingual named entity recognition in semi-structured text data. Our approach relies on both knowledge distillation and consistency training. The modeling framework leverages knowledge from a large language model (XLMRoBERTa) pre-trained on the source language, with a student-teacher relationship (knowledge distillation). The student model incorporates unsupervised consistency training (with KL divergence loss) on the low-resource target language. We employ two independent datasets of SMSs in English and Arabic, each carrying semi-structured banking transaction information, and focus on exhibiting the transfer of knowledge from English to Arabic. With access to only 30 labeled samples, our model can generalize the recognition of merchants, amounts, and other fields from English to Arabic. We show that our modeling approach, while efficient, performs best overall when compared to state-of-the-art approaches like DistilBERT pre-trained on the target language or a supervised model directly trained on labeled data in the target language. Our experiments show that it is enough to learn to recognize entities in English to reach reasonable performance in a low-resource language in the presence of a few labeled samples of semi-structured data. The proposed framework has implications for developing multi-lingual applications, especially in geographies where digital endeavors rely on both English and one or more low-resource language(s), sometimes mixed with English or employed singly.