Improving Representation Learning of Complex Critical Care Data with ICU-BERT

TL;DR

ICU-BERT is a transformer-based model pre-trained on ICU data that learns robust, generalizable representations, improving clinical decision support by effectively integrating complex multivariate and unstructured data.

Contribution

This paper introduces ICU-BERT, a novel transformer model pre-trained on ICU data with a multi-task scheme, enhancing representation learning for complex clinical datasets.

Findings

ICU-BERT outperforms existing benchmarks on multiple ICU tasks.

Incorporates dense embeddings from biomedical LLMs for richer data representation.

Demonstrates adaptability across diverse ICU datasets.

Abstract

The multivariate, asynchronous nature of real-world clinical data, such as that generated in Intensive Care Units (ICUs), challenges traditional AI-based decision-support systems. These often assume data regularity and feature independence and frequently rely on limited data scopes and manual feature engineering. The potential of generative AI technologies has not yet been fully exploited to analyze clinical data. We introduce ICU-BERT, a transformer-based model pre-trained on the MIMIC-IV database using a multi-task scheme to learn robust representations of complex ICU data with minimal preprocessing. ICU-BERT employs a multi-token input strategy, incorporating dense embeddings from a biomedical Large Language Model to learn a generalizable representation of complex and multivariate ICU data. With an initial evaluation of five tasks and four additional ICU datasets, ICU-BERT results indicate that ICU-BERT either compares to or surpasses current performance benchmarks by leveraging fine-tuning. By integrating structured and unstructured data, ICU-BERT advances the use of foundational models in medical informatics, offering an adaptable solution for clinical decision support across diverse applications.