CL-NERIL: A Cross-Lingual Model for NER in Indian Languages

TL;DR

This paper introduces CL-NERIL, a cross-lingual NER framework for Indian languages that leverages parallel corpora and a teacher-student model to improve performance in low-resource settings.

Contribution

It proposes a novel annotation projection method combined with a teacher-student model to enhance NER in Indian languages using weakly labeled data.

Findings

Minimum 10% performance improvement over zero-shot models

Effective use of weakly labeled data to supplement source language data

Framework applicable to multiple Indian languages

Abstract

Developing Named Entity Recognition (NER) systems for Indian languages has been a long-standing challenge, mainly owing to the requirement of a large amount of annotated clean training instances. This paper proposes an end-to-end framework for NER for Indian languages in a low-resource setting by exploiting parallel corpora of English and Indian languages and an English NER dataset. The proposed framework includes an annotation projection method that combines word alignment score and NER tag prediction confidence score on source language (English) data to generate weakly labeled data in a target Indian language. We employ a variant of the Teacher-Student model and optimize it jointly on the pseudo labels of the Teacher model and predictions on the generated weakly labeled data. We also present manually annotated test sets for three Indian languages: Hindi, Bengali, and Gujarati. We evaluate the performance of the proposed framework on the test sets of the three Indian languages. Empirical results show a minimum 10% performance improvement compared to the zero-shot transfer learning model on all languages. This indicates that weakly labeled data generated using the proposed annotation projection method in target Indian languages can complement well-annotated source language data to enhance performance. Our code is publicly available at https://github.com/aksh555/CL-NERIL