An Empirical Study of UMLS Concept Extraction from Clinical Notes using Boolean Combination Ensembles

TL;DR

This study evaluates how Boolean combinations of multiple NLP systems and filtering strategies impact UMLS concept extraction accuracy from clinical notes across diverse datasets.

Contribution

It introduces an empirical analysis of Boolean ensemble methods and filtering techniques to improve UMLS concept recognition in clinical NLP.

Findings

Boolean ensembling improved UMLS concept matching performance.

Grid-search optimization aids in balancing precision and recall.

Ensemble methods outperform individual systems in certain datasets.

Abstract

Our objective in this study is to investigate the behavior of Boolean operators on combining annotation output from multiple Natural Language Processing (NLP) systems across multiple corpora and to assess how filtering by aggregation of Unified Medical Language System (UMLS) Metathesaurus concepts affects system performance for Named Entity Recognition (NER) of UMLS concepts. We used three corpora annotated for UMLS concepts: 2010 i2b2 VA challenge set (31,161 annotations), Multi-source Integrated Platform for Answering Clinical Questions (MiPACQ) corpus (17,457 annotations including UMLS concept unique identifiers), and Fairview Health Services corpus (44,530 annotations). Our results showed that for UMLS concept matching, Boolean ensembling of the MiPACQ corpus trended towards higher performance over individual systems. Use of an approximate grid-search can help optimize the precision-recall tradeoff and can provide a set of heuristics for choosing an optimal set of ensembles.