MultiTab: A Scalable Foundation for Multitask Learning on Tabular Data

TL;DR

MultiTab introduces a scalable transformer-based multitask learning architecture tailored for large tabular datasets, demonstrating superior performance and providing a synthetic benchmark for systematic evaluation of multitask dynamics.

Contribution

It presents MultiTab-Net, a novel multitask transformer architecture with a masked-attention mechanism, and introduces MultiTab-Bench, a synthetic dataset generator for evaluating multitask learning in tabular data.

Findings

MultiTab-Net outperforms existing MTL models across diverse datasets.

The architecture effectively models feature dependencies and reduces task competition.

MultiTab-Bench enables systematic analysis of multitask learning behaviors.

Abstract

Tabular data is the most abundant data type in the world, powering systems in finance, healthcare, e-commerce, and beyond. As tabular datasets grow and span multiple related targets, there is an increasing need to exploit shared task information for improved multitask generalization. Multitask learning (MTL) has emerged as a powerful way to improve generalization and efficiency, yet most existing work focuses narrowly on large-scale recommendation systems, leaving its potential in broader tabular domains largely underexplored. Also, existing MTL approaches for tabular data predominantly rely on multi-layer perceptron-based backbones, which struggle to capture complex feature interactions and often fail to scale when data is abundant, a limitation that transformer architectures have overcome in other domains. Motivated by this, we introduce MultiTab-Net, the first multitask transformer architecture specifically designed for large tabular data. MultiTab-Net employs a novel multitask masked-attention mechanism that dynamically models feature-feature dependencies while mitigating task competition. Through extensive experiments, we show that MultiTab-Net consistently achieves higher multitask gain than existing MTL architectures and single-task transformers across diverse domains including large-scale recommendation data, census-like socioeconomic data, and physics datasets, spanning a wide range of task counts, task types, and feature modalities. In addition, we contribute MultiTab-Bench, a generalized multitask synthetic dataset generator that enables systematic evaluation of multitask dynamics by tuning task count, task correlations, and relative task complexity. Our code is publicly available at https://github.com/Armanfard-Lab/MultiTab.