Robustness is Important: Limitations of LLMs for Data Fitting

TL;DR

This paper reveals that large language models are highly sensitive to irrelevant data representation changes, such as variable renaming, which significantly impacts their prediction accuracy, raising concerns about their robustness for data fitting tasks.

Contribution

The study uncovers the vulnerability of LLMs to task-irrelevant variations and analyzes attention patterns, highlighting limitations even in specialized models like TabPFN.

Findings

Prediction error can vary by up to 82% due to irrelevant data changes.

Both in-context learning and fine-tuning exhibit sensitivity to data representation.

State-of-the-art tabular models like TabPFN are also affected by irrelevant variations.

Abstract

Large Language Models (LLMs) are being applied in a wide array of settings, well beyond the typical language-oriented use cases. In particular, LLMs are increasingly used as a plug-and-play method for fitting data and generating predictions. Prior work has shown that LLMs, via in-context learning or supervised fine-tuning, can perform competitively with many tabular supervised learning techniques in terms of predictive performance. However, we identify a critical vulnerability of using LLMs for data fitting -- making changes to data representation that are completely irrelevant to the underlying learning task can drastically alter LLMs' predictions on the same data. For example, simply changing variable names can sway the size of prediction error by as much as 82% in certain settings. Such prediction sensitivity with respect to task-irrelevant variations manifests under both in-context learning and supervised fine-tuning, for both close-weight and open-weight general-purpose LLMs. Moreover, by examining the attention scores of an open-weight LLM, we discover a non-uniform attention pattern: training examples and variable names/values which happen to occupy certain positions in the prompt receive more attention when output tokens are generated, even though different positions are expected to receive roughly the same attention. This partially explains the sensitivity in the presence of task-irrelevant variations. We also consider a state-of-the-art tabular foundation model (TabPFN) trained specifically for data fitting. Despite being explicitly designed to achieve prediction robustness, TabPFN is still not immune to task-irrelevant variations. Overall, despite LLMs' impressive predictive capabilities, currently they lack even the basic level of robustness to be used as a principled data-fitting tool.