SoK: Challenges in Tabular Membership Inference Attacks

TL;DR

This paper reviews and analyzes membership inference attacks on tabular data, revealing their limited effectiveness overall but highlighting vulnerabilities of unique records and the impact of surrogate models, especially in federated learning scenarios.

Contribution

It provides an extensive taxonomy and analysis of MIAs for tabular data, including new insights into attack transferability, defenses, and vulnerabilities of single-out records in federated learning.

Findings

MIAs show generally poor performance on tabular data

Single-out records are highly vulnerable to MIAs

Using different surrogate models enhances attack effectiveness

Abstract

Membership Inference Attacks (MIAs) are currently a dominant approach for evaluating privacy in machine learning applications. Despite their significance in identifying records belonging to the training dataset, several concerns remain unexplored, particularly with regard to tabular data. In this paper, first, we provide an extensive review and analysis of MIAs considering two main learning paradigms: centralized and federated learning. We extend and refine the taxonomy for both. Second, we demonstrate the efficacy of MIAs in tabular data using several attack strategies, also including defenses. Furthermore, in a federated learning scenario, we consider the threat posed by an outsider adversary, which is often neglected. Third, we demonstrate the high vulnerability of single-outs (records with a unique signature) to MIAs. Lastly, we explore how MIAs transfer across model architectures. Our results point towards a general poor performance of these attacks in tabular data which contrasts with previous state-of-the-art. Notably, even attacks with limited attack performance can still successfully expose a large portion of single-outs. Moreover, our findings suggest that using different surrogate models makes MIAs more effective.