TabText: Language-Based Representations of Tabular Health Data for Predictive Modelling

TL;DR

TabText introduces a novel method that converts tabular medical data into contextual language representations using pretrained language models, improving predictive performance and generalization across diverse healthcare datasets.

Contribution

The paper presents TabText, a new approach that leverages language models for feature extraction from tabular health data, reducing manual preprocessing and enhancing model accuracy.

Findings

Achieved AUC of 0.75-0.94 on inpatient tasks across hospitals.

Improved out-of-sample AUC by up to 4 percentage points with embeddings.

Generalized well to unseen hospitals and datasets.

Abstract

Tabular medical records remain the most readily available data format for applying machine learning in healthcare. However, traditional data preprocessing ignores valuable contextual information in tables and requires substantial manual cleaning and harmonisation, creating a bottleneck for model development. We introduce TabText, a preprocessing and feature extraction method that leverages contextual information and streamlines the curation of tabular medical data. This method converts tables into contextual language and applies pretrained large language models (LLMs) to generate task-independent numerical representations. These fixed embeddings are then used as input for various predictive tasks. TabText was evaluated on nine inpatient flow prediction tasks (e.g., ICU admission, discharge, mortality) using electronic medical records across six hospitals from a US health system, and on nine publicly available datasets from the UCI Machine Learning Repository, covering tasks such as cancer diagnosis, recurrence, and survival. TabText models trained on unprocessed data from a single hospital (572,964 patient-days, Jan 2018-Dec 2020) achieved accurate performance (AUC 0.75-0.94) when tested prospectively on 265,917 patient-days from Jan 2021-Apr 2022, and generalised well to five additional hospitals not used for training. When augmenting preprocessed tabular records with these contextual embeddings, out-of-sample AUC improved by up to 4 additive percentage points in challenging tasks such as ICU transfer and breast cancer recurrence, while providing little to no benefit for already high-performing tasks. Findings were consistent across both private and public datasets.