Converse Attention Knowledge Transfer for Low-Resource Named Entity Recognition

TL;DR

This paper introduces Converse Attention Network (CAN), a novel method that leverages high-resource English models and attention-based translation to improve NER performance in low-resource languages.

Contribution

The paper proposes CAN, a new approach that aligns low-resource language features with high-resource English models using attention matrices for better NER results.

Findings

CAN achieves significant improvements on four low-resource NER datasets.

The method effectively aligns semantic features across languages.

Results demonstrate the potential of attention-based transfer for low-resource NLP tasks.

Abstract

In recent years, great success has been achieved in many tasks of natural language processing (NLP), e.g., named entity recognition (NER), especially in the high-resource language, i.e., English, thanks in part to the considerable amount of labeled resources. However, most low-resource languages do not have such an abundance of labeled data as high-resource English, leading to poor performance of NER in these low-resource languages. Inspired by knowledge transfer, we propose Converse Attention Network, or CAN in short, to improve the performance of NER in low-resource languages by leveraging the knowledge learned in pretrained high-resource English models. CAN first translates low-resource languages into high-resource English using an attention based translation module. In the process of translation, CAN obtain the attention matrices that align the two languages. Furthermore, CAN use the attention matrices to align the high-resource semantic features from a pretrained high-resource English model with the low-resource semantic features. As a result, CAN obtains aligned high-resource semantic features to enrich the representations of low-resource languages. Experiments on four low-resource NER datasets show that CAN achieves consistent and significant performance improvements, which indicates the effectiveness of CAN.