Multi-branch of Attention Yields Accurate Results for Tabular Data

TL;DR

MAYA introduces a multi-branch attention mechanism within a transformer framework to better handle feature heterogeneity in tabular data, achieving superior classification and regression results.

Contribution

The paper proposes MAYA, a novel transformer-based framework with multi-branch attention and collaborative learning for improved tabular data modeling.

Findings

Outperforms existing transformer-based methods on various datasets.

Effectively fuses heterogeneous features with limited parameter increase.

Achieves state-of-the-art results in tabular classification and regression.

Abstract

Tabular data inherently exhibits significant feature heterogeneity, but existing transformer-based methods lack specialized mechanisms to handle this property. To bridge the gap, we propose MAYA, an encoder-decoder transformer-based framework. In the encoder, we design a Multi-Branch of Attention (MBA) that constructs multiple parallel attention branches and averages the features at each branch, effectively fusing heterogeneous features while limiting parameter growth. Additionally, we employ collaborative learning with a dynamic consistency weight constraint to produce more robust representations. In the decoder stage, cross-attention is utilized to seamlessly integrate tabular data with corresponding label features. This dual-attention mechanism effectively captures both intra-instance and inter-instance interactions. We evaluate the proposed method on a wide range of datasets and compare it with other state-of-the-art transformer-based methods. Extensive experiments demonstrate that our model achieves superior performance among transformer-based methods in both tabular classification and regression tasks.