SFE-AI at SemEval-2022 Task 11: Low-Resource Named Entity Recognition using Large Pre-trained Language Models

TL;DR

This paper presents an adaptive ensemble method using a Transformer layer to combine multiple large pre-trained language models for low-resource named entity recognition, improving performance in open domain settings.

Contribution

It introduces a novel adaptive ensemble approach with a Transformer layer to leverage diverse models for NER in low-resource and open domain scenarios.

Findings

Achieved superior performance in Farsi and Dutch NER tasks.

Effective model integration through input-dependent weighting.

Enhanced differentiation of model advantages in ensemble.

Abstract

Large scale pre-training models have been widely used in named entity recognition (NER) tasks. However, model ensemble through parameter averaging or voting can not give full play to the differentiation advantages of different models, especially in the open domain. This paper describes our NER system in the SemEval 2022 task11: MultiCoNER. We proposed an effective system to adaptively ensemble pre-trained language models by a Transformer layer. By assigning different weights to each model for different inputs, we adopted the Transformer layer to integrate the advantages of diverse models effectively. Experimental results show that our method achieves superior performances in Farsi and Dutch.