When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?

TL;DR

This paper investigates whether large language model embeddings can effectively represent numerical medical data for machine learning tasks, comparing their performance to raw data in clinical prediction models.

Contribution

It evaluates the effectiveness of zero-shot LLM embeddings as feature extractors for medical diagnostics, highlighting their competitive performance against raw numerical data.

Findings

Raw data features still outperform LLM embeddings in medical ML tasks.

Zero-shot LLM embeddings show promising results as feature representations.

Prompt engineering impacts the utility of LLM embeddings in clinical predictions.

Abstract

The introduction of Large Language Models (LLMs) has advanced data representation and analysis, bringing significant progress in their use for medical questions and answering. Despite these advancements, integrating tabular data, especially numerical data pivotal in clinical contexts, into LLM paradigms has not been thoroughly explored. In this study, we examine the effectiveness of vector representations from last hidden states of LLMs for medical diagnostics and prognostics using electronic health record (EHR) data. We compare the performance of these embeddings with that of raw numerical EHR data when used as feature inputs to traditional machine learning (ML) algorithms that excel at tabular data learning, such as eXtreme Gradient Boosting. We focus on instruction-tuned LLMs in a zero-shot setting to represent abnormal physiological data and evaluating their utilities as feature extractors to enhance ML classifiers for predicting diagnoses, length of stay, and mortality. Furthermore, we examine prompt engineering techniques on zero-shot and few-shot LLM embeddings to measure their impact comprehensively. Although findings suggest the raw data features still prevails in medical ML tasks, zero-shot LLM embeddings demonstrate competitive results, suggesting a promising avenue for future research in medical applications.