Transformers with Stochastic Competition for Tabular Data Modelling

TL;DR

This paper introduces a novel Transformer-based deep learning model with stochastic competition mechanisms tailored for tabular data, achieving high performance and advancing the application of neural networks in this domain.

Contribution

The paper presents a new stochastic deep learning model for tabular data that incorporates specialized architectural modifications and stochastic competition mechanisms, improving performance over existing methods.

Findings

Model outperforms traditional methods on public datasets

Stochastic units enhance generalization and sparsity

Novel embedding layer improves feature representation

Abstract

Despite the prevalence and significance of tabular data across numerous industries and fields, it has been relatively underexplored in the realm of deep learning. Even today, neural networks are often overshadowed by techniques such as gradient boosted decision trees (GBDT). However, recent models are beginning to close this gap, outperforming GBDT in various setups and garnering increased attention in the field. Inspired by this development, we introduce a novel stochastic deep learning model specifically designed for tabular data. The foundation of this model is a Transformer-based architecture, carefully adapted to cater to the unique properties of tabular data through strategic architectural modifications and leveraging two forms of stochastic competition. First, we employ stochastic "Local Winner Takes All" units to promote generalization capacity through stochasticity and sparsity. Second, we introduce a novel embedding layer that selects among alternative linear embedding layers through a mechanism of stochastic competition. The effectiveness of the model is validated on a variety of widely-used, publicly available datasets. We demonstrate that, through the incorporation of these elements, our model yields high performance and marks a significant advancement in the application of deep learning to tabular data.