Tab-PET: Graph-Based Positional Encodings for Tabular Transformers

TL;DR

This paper introduces Tab-PET, a graph-based method for incorporating positional encodings into tabular transformers, which improves their generalization by reducing feature dimensionality and leveraging structural cues.

Contribution

The paper proposes a novel graph-based framework for estimating and integrating positional encodings into tabular transformers, demonstrating significant performance gains across multiple datasets.

Findings

Graph-derived PEs improve transformer performance on tabular data.

Association-based graphs outperform causality-based graphs in stability and gains.

Positional encodings reduce the effective rank of features, aiding generalization.

Abstract

Supervised learning with tabular data presents unique challenges, including low data sizes, the absence of structural cues, and heterogeneous features spanning both categorical and continuous domains. Unlike vision and language tasks, where models can exploit inductive biases in the data, tabular data lacks inherent positional structure, hindering the effectiveness of self-attention mechanisms. While recent transformer-based models like TabTransformer, SAINT, and FT-Transformer (which we refer to as 3T) have shown promise on tabular data, they typically operate without leveraging structural cues such as positional encodings (PEs), as no prior structural information is usually available. In this work, we find both theoretically and empirically that structural cues, specifically PEs can be a useful tool to improve generalization performance for tabular transformers. We find that PEs impart the ability to reduce the effective rank (a form of intrinsic dimensionality) of the features, effectively simplifying the task by reducing the dimensionality of the problem, yielding improved generalization. To that end, we propose Tab-PET (PEs for Tabular Transformers), a graph-based framework for estimating and inculcating PEs into embeddings. Inspired by approaches that derive PEs from graph topology, we explore two paradigms for graph estimation: association-based and causality-based. We empirically demonstrate that graph-derived PEs significantly improve performance across 50 classification and regression datasets for 3T. Notably, association-based graphs consistently yield more stable and pronounced gains compared to causality-driven ones. Our work highlights an unexpected role of PEs in tabular transformers, revealing how they can be harnessed to improve generalization.