Cost-Quality Adaptive Active Learning for Chinese Clinical Named Entity Recognition

TL;DR

This paper introduces a novel cost-quality adaptive active learning method for Chinese clinical named entity recognition, optimizing annotation efficiency by balancing quality, cost, and informativeness in a multi-labeler setting.

Contribution

It proposes a new active learning approach that adaptively selects instance-labeler pairs considering cost and quality, improving annotation efficiency for Chinese EHRs.

Findings

Outperforms baseline methods on CCKS-2017 dataset

Achieves higher annotation quality at lower costs

Demonstrates effectiveness in real-world clinical data annotation

Abstract

Clinical Named Entity Recognition (CNER) aims to automatically identity clinical terminologies in Electronic Health Records (EHRs), which is a fundamental and crucial step for clinical research. To train a high-performance model for CNER, it usually requires a large number of EHRs with high-quality labels. However, labeling EHRs, especially Chinese EHRs, is time-consuming and expensive. One effective solution to this is active learning, where a model asks labelers to annotate data which the model is uncertain of. Conventional active learning assumes a single labeler that always replies noiseless answers to queried labels. However, in real settings, multiple labelers provide diverse quality of annotation with varied costs and labelers with low overall annotation quality can still assign correct labels for some specific instances. In this paper, we propose a Cost-Quality Adaptive Active Learning (CQAAL) approach for CNER in Chinese EHRs, which maintains a balance between the annotation quality, labeling costs, and the informativeness of selected instances. Specifically, CQAAL selects cost-effective instance-labeler pairs to achieve better annotation quality with lower costs in an adaptive manner. Computational results on the CCKS-2017 Task 2 benchmark dataset demonstrate the superiority and effectiveness of the proposed CQAAL.