Fine-Tuning the Retrieval Mechanism for Tabular Deep Learning

TL;DR

This paper investigates a retrieval-based training approach for tabular deep learning, demonstrating that fine-tuning pretrained models with retrieval mechanisms significantly improves performance over traditional methods.

Contribution

It introduces a retrieval mechanism for neural networks in tabular data, combined with fine-tuning and pretraining, to enhance predictive accuracy beyond existing models.

Findings

Retrieval-based training outperforms traditional methods.

Fine-tuning pretrained models boosts performance.

Pretraining is crucial for effective retrieval-based learning.

Abstract

While interests in tabular deep learning has significantly grown, conventional tree-based models still outperform deep learning methods. To narrow this performance gap, we explore the innovative retrieval mechanism, a methodology that allows neural networks to refer to other data points while making predictions. Our experiments reveal that retrieval-based training, especially when fine-tuning the pretrained TabPFN model, notably surpasses existing methods. Moreover, the extensive pretraining plays a crucial role to enhance the performance of the model. These insights imply that blending the retrieval mechanism with pretraining and transfer learning schemes offers considerable potential for advancing the field of tabular deep learning.