Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learning

TL;DR

Latte is a training-time framework that transfers latent knowledge from Large Language Models to improve few-shot tabular learning, reducing latency and enhancing performance with limited labeled data.

Contribution

Latte introduces a novel training-time knowledge transfer method that leverages LLMs' latent knowledge for better few-shot tabular learning, overcoming existing limitations.

Findings

Latte outperforms existing methods on multiple benchmarks.

It effectively utilizes unlabeled data to improve performance.

Latte reduces overfitting in few-shot scenarios.

Abstract

Few-shot tabular learning, in which machine learning models are trained with a limited amount of labeled data, provides a cost-effective approach to addressing real-world challenges. The advent of Large Language Models (LLMs) has sparked interest in leveraging their pre-trained knowledge for few-shot tabular learning. Despite promising results, existing approaches either rely on test-time knowledge extraction, which introduces undesirable latency, or text-level knowledge, which leads to unreliable feature engineering. To overcome these limitations, we propose Latte, a training-time knowledge extraction framework that transfers the latent prior knowledge within LLMs to optimize a more generalized downstream model. Latte enables general knowledge-guided downstream tabular learning, facilitating the weighted fusion of information across different feature values while reducing the risk of overfitting to limited labeled data. Furthermore, Latte is compatible with existing unsupervised pre-training paradigms and effectively utilizes available unlabeled samples to overcome the performance limitations imposed by an extremely small labeled dataset. Extensive experiments on various few-shot tabular learning benchmarks demonstrate the superior performance of Latte, establishing it as a state-of-the-art approach in this domain