Closing the gap on tabular data with Fourier and Implicit Categorical Features

TL;DR

This paper introduces a novel approach combining statistical feature processing and Learned Fourier techniques to enhance deep learning performance on tabular data, bridging the gap with tree-based methods.

Contribution

It proposes a new preprocessing method and Fourier-based bias mitigation to improve neural network effectiveness on tabular datasets, outperforming traditional deep models.

Findings

Deep models with proposed preprocessing outperform baseline neural networks.

Achieves performance comparable or superior to XGBoost on benchmark datasets.

Significantly reduces the performance gap between neural networks and tree-based methods.

Abstract

While Deep Learning has demonstrated impressive results in applications on various data types, it continues to lag behind tree-based methods when applied to tabular data, often referred to as the last "unconquered castle" for neural networks. We hypothesize that a significant advantage of tree-based methods lies in their intrinsic capability to model and exploit non-linear interactions induced by features with categorical characteristics. In contrast, neural-based methods exhibit biases toward uniform numerical processing of features and smooth solutions, making it challenging for them to effectively leverage such patterns. We address this performance gap by using statistical-based feature processing techniques to identify features that are strongly correlated with the target once discretized. We further mitigate the bias of deep models for overly-smooth solutions, a bias that does not align with the inherent properties of the data, using Learned Fourier. We show that our proposed feature preprocessing significantly boosts the performance of deep learning models and enables them to achieve a performance that closely matches or surpasses XGBoost on a comprehensive tabular data benchmark.