Adapting Pretrained Language Models for Solving Tabular Prediction Problems in the Electronic Health Record

TL;DR

This paper adapts a pretrained DeBERTa model for electronic health record prediction tasks, demonstrating improved performance over baseline models through domain-specific pretraining and data processing techniques.

Contribution

It introduces a domain-adapted DeBERTa model for EHR tasks and compares its effectiveness with other models, highlighting the impact of data preprocessing.

Findings

Proposed model outperforms alternatives on two of three tasks (p<0.001)

Matching performance on the third task

Descriptive columns improve model performance

Abstract

We propose an approach for adapting the DeBERTa model for electronic health record (EHR) tasks using domain adaptation. We pretrain a small DeBERTa model on a dataset consisting of MIMIC-III discharge summaries, clinical notes, radiology reports, and PubMed abstracts. We compare this model's performance with a DeBERTa model pre-trained on clinical texts from our institutional EHR (MeDeBERTa) and an XGBoost model. We evaluate performance on three benchmark tasks for emergency department outcomes using the MIMIC-IV-ED dataset. We preprocess the data to convert it into text format and generate four versions of the original datasets to compare data processing and data inclusion. The results show that our proposed approach outperforms the alternative models on two of three tasks (p<0.001) and matches performance on the third task, with the use of descriptive columns improving performance over the original column names.