Revisiting Pretraining Objectives for Tabular Deep Learning

TL;DR

This paper investigates various pretraining strategies for tabular deep learning models, demonstrating that proper pretraining, especially with target-aware objectives, can significantly enhance performance and surpass traditional GBDT models.

Contribution

It identifies effective pretraining practices for tabular DL models, highlighting the benefits of target-aware objectives and providing comprehensive comparisons across architectures.

Findings

Pretraining with target labels improves downstream performance.

Proper pretraining can make tabular DL models outperform GBDTs.

Target-aware pretraining objectives are particularly beneficial.

Abstract

Recent deep learning models for tabular data currently compete with the traditional ML models based on decision trees (GBDT). Unlike GBDT, deep models can additionally benefit from pretraining, which is a workhorse of DL for vision and NLP. For tabular problems, several pretraining methods were proposed, but it is not entirely clear if pretraining provides consistent noticeable improvements and what method should be used, since the methods are often not compared to each other or comparison is limited to the simplest MLP architectures. In this work, we aim to identify the best practices to pretrain tabular DL models that can be universally applied to different datasets and architectures. Among our findings, we show that using the object target labels during the pretraining stage is beneficial for the downstream performance and advocate several target-aware pretraining objectives. Overall, our experiments demonstrate that properly performed pretraining significantly increases the performance of tabular DL models, which often leads to their superiority over GBDTs.