VisTabNet: Adapting Vision Transformers for Tabular Data

TL;DR

VisTabNet introduces a novel approach to adapt pre-trained Vision Transformers for tabular data by projecting tabular inputs into patch embeddings, enabling effective transfer learning and outperforming traditional methods on small datasets.

Contribution

The paper presents a new transfer learning method that adapts Vision Transformers for tabular data, reducing architecture design effort and improving performance on small datasets.

Findings

Outperforms traditional ensemble and deep learning models on small tabular datasets.

Demonstrates the feasibility of transferring pre-trained image models to tabular data tasks.

Shows that cross-modal transfer learning can extend the applicability of Vision Transformers.

Abstract

Although deep learning models have had great success in natural language processing and computer vision, we do not observe comparable improvements in the case of tabular data, which is still the most common data type used in biological, industrial and financial applications. In particular, it is challenging to transfer large-scale pre-trained models to downstream tasks defined on small tabular datasets. To address this, we propose VisTabNet -- a cross-modal transfer learning method, which allows for adapting Vision Transformer (ViT) with pre-trained weights to process tabular data. By projecting tabular inputs to patch embeddings acceptable by ViT, we can directly apply a pre-trained Transformer Encoder to tabular inputs. This approach eliminates the conceptual cost of designing a suitable architecture for processing tabular data, while reducing the computational cost of training the model from scratch. Experimental results on multiple small tabular datasets (less than 1k samples) demonstrate VisTabNet's superiority, outperforming both traditional ensemble methods and recent deep learning models. The proposed method goes beyond conventional transfer learning practice and shows that pre-trained image models can be transferred to solve tabular problems, extending the boundaries of transfer learning. We share our example implementation as a GitHub repository available at https://github.com/wwydmanski/VisTabNet.