Cross-institution text mining to uncover clinical associations: a case study relating social factors and code status in intensive care medicine

TL;DR

This study evaluates the use of off-the-shelf and adapted text mining models to extract social factors from clinical notes in intensive care, assessing their reliability in association studies with code status.

Contribution

It investigates the effectiveness of external and adapted text mining models for extracting social factors in ICU records, highlighting limitations and the need for better models.

Findings

External models improved F1-scores but did not change associations significantly.

Current models are unreliable for accurate association analysis.

Further research and larger labeled datasets are needed for reliable text mining in medical studies.

Abstract

Objective: Text mining of clinical notes embedded in electronic medical records is increasingly used to extract patient characteristics otherwise not or only partly available, to assess their association with relevant health outcomes. As manual data labeling needed to develop text mining models is resource intensive, we investigated whether off-the-shelf text mining models developed at external institutions, together with limited within-institution labeled data, could be used to reliably extract study variables to conduct association studies. Materials and Methods: We developed multiple text mining models on different combinations of within-institution and external-institution data to extract social factors from discharge reports of intensive care patients. Subsequently, we assessed the associations between social factors and having a do-not-resuscitate/intubate code. Results: Important differences were found between associations based on manually labeled data compared to text-mined social factors in three out of five cases. Adopting external-institution text mining models using manually labeled within-institution data resulted in models with higher F1-scores, but not in meaningfully different associations. Discussion: While text mining facilitated scaling analyses to larger samples leading to discovering a larger number of associations, the estimates may be unreliable. Confirmation is needed with better text mining models, ideally on a larger manually labeled dataset. Conclusion: The currently used text mining models were not sufficiently accurate to be used reliably in an association study. Model adaptation using within-institution data did not improve the estimates. Further research is needed to set conditions for reliable use of text mining in medical research.