One Model to Recognize Them All: Marginal Distillation from NER Models with Different Tag Sets

TL;DR

This paper introduces MARDI, a flexible method for training a unified NER model from heterogeneous resources using only pre-trained models, improving integration and performance across diverse entity types and architectures.

Contribution

MARDI is a novel marginal distillation approach that requires only pre-trained models, enabling unified NER training across different tag sets without access to original datasets.

Findings

MARDI matches strong CRF baselines in performance.

MARDI achieves state-of-the-art on progressive NER.

MARDI is flexible across various NER architectures.

Abstract

Named entity recognition (NER) is a fundamental component in the modern language understanding pipeline. Public NER resources such as annotated data and model services are available in many domains. However, given a particular downstream application, there is often no single NER resource that supports all the desired entity types, so users must leverage multiple resources with different tag sets. This paper presents a marginal distillation (MARDI) approach for training a unified NER model from resources with disjoint or heterogeneous tag sets. In contrast to recent works, MARDI merely requires access to pre-trained models rather than the original training datasets. This flexibility makes it easier to work with sensitive domains like healthcare and finance. Furthermore, our approach is general enough to integrate with different NER architectures, including local models (e.g., BiLSTM) and global models (e.g., CRF). Experiments on two benchmark datasets show that MARDI performs on par with a strong marginal CRF baseline, while being more flexible in the form of required NER resources. MARDI also sets a new state of the art on the progressive NER task. MARDI significantly outperforms the start-of-the-art model on the task of progressive NER.