An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotyping

TL;DR

This study evaluates the effectiveness of clinical BERT models in diabetic eye disease phenotyping, revealing that out-of-distribution models perform comparably to domain-specific models, challenging assumptions about clinical data domain necessity.

Contribution

The paper introduces a system for extracting clinical concepts related to diabetic eye disease and provides a comprehensive evaluation of clinical BERT models across different training paradigms.

Findings

Out-of-distribution BERT models perform similarly to clinical domain-specific models.

Clinical language models pretrained on clinical data do not significantly outperform non-clinical models.

Clinical language data should not be treated as a single homogeneous domain.

Abstract

Diabetic eye disease is a major cause of blindness worldwide. The ability to monitor relevant clinical trajectories and detect lapses in care is critical to managing the disease and preventing blindness. Alas, much of the information necessary to support these goals is found only in the free text of the electronic medical record. To fill this information gap, we introduce a system for extracting evidence from clinical text of 19 clinical concepts related to diabetic eye disease and inferring relevant attributes for each. In developing this ophthalmology phenotyping system, we are also afforded a unique opportunity to evaluate the effectiveness of clinical language models at adapting to new clinical domains. Across multiple training paradigms, we find that BERT language models pretrained on out-of-distribution clinical data offer no significant improvement over BERT language models pretrained on non-clinical data for our domain. Our study tempers recent claims that language models pretrained on clinical data are necessary for clinical NLP tasks and highlights the importance of not treating clinical language data as a single homogeneous domain.