Towards DS-NER: Unveiling and Addressing Latent Noise in Distant Annotations

TL;DR

This paper investigates the latent noise in distantly supervised NER, proposing a novel noise assessment framework and methods that improve robustness across various datasets and annotation techniques.

Contribution

It introduces a new framework for noise assessment in DS-NER, addressing unlabeled and noisy entity problems with tailored solutions, and demonstrates significant performance improvements.

Findings

Achieved superior results on eight real-world datasets.

Effectively distinguishes between unlabeled and noisy entity issues.

Enhances robustness of DS-NER models across diverse sources.

Abstract

Distantly supervised named entity recognition (DS-NER) has emerged as a cheap and convenient alternative to traditional human annotation methods, enabling the automatic generation of training data by aligning text with external resources. Despite the many efforts in noise measurement methods, few works focus on the latent noise distribution between different distant annotation methods. In this work, we explore the effectiveness and robustness of DS-NER by two aspects: (1) distant annotation techniques, which encompasses both traditional rule-based methods and the innovative large language model supervision approach, and (2) noise assessment, for which we introduce a novel framework. This framework addresses the challenges by distinctly categorizing them into the unlabeled-entity problem (UEP) and the noisy-entity problem (NEP), subsequently providing specialized solutions for each. Our proposed method achieves significant improvements on eight real-world distant supervision datasets originating from three different data sources and involving four distinct annotation techniques, confirming its superiority over current state-of-the-art methods.