Preserving Empirical Probabilities in BERT for Small-sample Clinical Entity Recognition

TL;DR

This paper investigates how to improve BERT-based clinical entity recognition in small-sample, imbalanced datasets by analyzing loss mechanisms and proposing methods to preserve empirical probabilities for better minority class recognition.

Contribution

It introduces novel techniques to address label imbalance in BERT for clinical NER, enhancing recognition of underrepresented entity types.

Findings

Improved recognition accuracy for minority entity classes.

Analysis of loss mechanisms reveals key factors affecting imbalance.

Proposed methods outperform baseline models in imbalanced settings.

Abstract

Named Entity Recognition (NER) encounters the challenge of unbalanced labels, where certain entity types are overrepresented while others are underrepresented in real-world datasets. This imbalance can lead to biased models that perform poorly on minority entity classes, impeding accurate and equitable entity recognition. This paper explores the effects of unbalanced entity labels of the BERT-based pre-trained model. We analyze the different mechanisms of loss calculation and loss propagation for the task of token classification on randomized datasets. Then we propose ways to improve the token classification for the highly imbalanced task of clinical entity recognition.