TabAttackBench: A Benchmark for Adversarial Attacks on Tabular Data

TL;DR

This paper introduces TabAttackBench, a comprehensive benchmark for evaluating adversarial attacks on tabular data, analyzing effectiveness and imperceptibility across multiple models, datasets, and attack algorithms.

Contribution

It provides the first extensive benchmark assessing various white-box adversarial attacks on tabular datasets with multiple metrics for imperceptibility.

Findings

$ ext{l}_ ext{infty}$-based attacks are more successful but less subtle.

$ ext{l}_ ext{2}$-based attacks produce more realistic perturbations.

Trade-offs exist between attack success and imperceptibility.

Abstract

Adversarial attacks pose a significant threat to machine learning models by inducing incorrect predictions through imperceptible perturbations to input data. While these attacks are well studied in unstructured domains such as images, their behaviour on tabular data remains underexplored due to mixed feature types and complex inter-feature dependencies. This study introduces a comprehensive benchmark that evaluates adversarial attacks on tabular datasets with respect to both effectiveness and imperceptibility. We assess five white-box attack algorithms (FGSM, BIM, PGD, DeepFool, and C\&W) across four representative models (LR, MLP, TabTransformer and FT-Transformer) using eleven datasets spanning finance, energy, and healthcare domains. The benchmark employs four quantitative imperceptibility metrics (proximity, sparsity, deviation, and sensitivity) to characterise perturbation realism. The analysis quantifies the trade-off between these two aspects and reveals consistent differences between attack types, with $\ell_\infty$-based attacks achieving higher success but lower subtlety, and $\ell_2$-based attacks offering more realistic perturbations. The benchmark findings offer actionable insights for designing more imperceptible adversarial attacks, advancing the understanding of adversarial vulnerability in tabular machine learning.