LLM Embeddings Improve Test-time Adaptation to Tabular $Y|X$-Shifts

TL;DR

This paper explores how leveraging large language model (LLM) embeddings can improve the robustness and adaptability of models to $Y|X$-shifts in tabular data, enabling effective domain adaptation with minimal labeled data.

Contribution

It introduces a method of using LLM embeddings for tabular data to enhance test-time adaptation to $Y|X$-shifts, supported by extensive empirical evaluation.

Findings

LLM embeddings alone offer inconsistent robustness improvements.

Models trained on LLM embeddings can be effectively fine-tuned with as few as 32 labeled samples.

The approach is validated across 7650 source-target pairs and 261,000 model configurations.

Abstract

For tabular datasets, the change in the relationship between the label and covariates ($Y|X$-shifts) is common due to missing variables (a.k.a. confounders). Since it is impossible to generalize to a completely new and unknown domain, we study models that are easy to adapt to the target domain even with few labeled examples. We focus on building more informative representations of tabular data that can mitigate $Y|X$-shifts, and propose to leverage the prior world knowledge in LLMs by serializing (write down) the tabular data to encode it. We find LLM embeddings alone provide inconsistent improvements in robustness, but models trained on them can be well adapted/finetuned to the target domain even using 32 labeled observations. Our finding is based on a comprehensive and systematic study consisting of 7650 source-target pairs and benchmark against 261,000 model configurations trained by 22 algorithms. Our observation holds when ablating the size of accessible target data and different adaptation strategies. The code is available at https://github.com/namkoong-lab/LLM-Tabular-Shifts.