LATTLE: LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains

TL;DR

LATTLE introduces a novel attention transplant method enabling large language models to transfer knowledge across disparate tabular datasets without shared features or prompt engineering, improving cross-domain transfer learning in low-resource settings.

Contribution

The paper proposes a domain-agnostic attention transplant mechanism that allows LLMs to effectively transfer learning across different tabular data domains without shared features or extensive prompt engineering.

Findings

Outperforms traditional ML and deep tabular models

Effective in low-resource, cross-domain scenarios

Demonstrates superiority over models trained on large datasets

Abstract

Transfer learning on tabular data is challenging due to disparate feature spaces across domains, in contrast to the homogeneous structures of image and text. Large language models (LLMs) offer a knowledge base to improve the limited effectiveness of cross-domain transfer learning for tabular data. However, LLM performance often stagnates due to subjective text prompts and the computational limitations of in-context learning. We present a novel language-to-tabular context-learning method that uses attention-specific transformer weights, enabling seamless transfer learning across disparate tabular data sets. The LLM attention transplant mechanism facilitates a domain-agnostic transfer learning, eliminating the need for shared features between tables, LLM prompt engineering, and large-scale pretrained models. Our experiments using ten pairs of disjoint source-target data sets and 12 baseline methods demonstrate the superiority of the proposed LLM-attention transplant for transfer learning (LATTLE) method over traditional ML models, state-of-the-art deep tabular architectures, and models trained on thousands to billions of tabular samples. The proposed cross-domain attention transfer demonstrates an effective solution for adapting LLMs to learning non-text tabular data in a low-resource environment. The source code of the LATTLE implementation is publicly available.