Clinical Language Understanding Evaluation (CLUE)

TL;DR

The CLUE benchmark provides standardized datasets and evaluation tools for clinical language understanding tasks, aiming to improve reproducibility and comparability of models in clinical NLP.

Contribution

This paper introduces the CLUE benchmark with four clinical language tasks, standardized datasets, and a software toolkit to facilitate fair comparison and reproducibility.

Findings

Provides a unified benchmark for clinical NLP tasks

Enables direct comparison of different models

Aims to accelerate development of clinical language understanding methods

Abstract

Clinical language processing has received a lot of attention in recent years, resulting in new models or methods for disease phenotyping, mortality prediction, and other tasks. Unfortunately, many of these approaches are tested under different experimental settings (e.g., data sources, training and testing splits, metrics, evaluation criteria, etc.) making it difficult to compare approaches and determine state-of-the-art. To address these issues and facilitate reproducibility and comparison, we present the Clinical Language Understanding Evaluation (CLUE) benchmark with a set of four clinical language understanding tasks, standard training, development, validation and testing sets derived from MIMIC data, as well as a software toolkit. It is our hope that these data will enable direct comparison between approaches, improve reproducibility, and reduce the barrier-to-entry for developing novel models or methods for these clinical language understanding tasks.