MIDST Challenge at SaTML 2025: Membership Inference over Diffusion-models-based Synthetic Tabular data

TL;DR

The paper evaluates the privacy resilience of diffusion-model-generated synthetic tabular data against membership inference attacks, introducing novel attack methods and providing a comprehensive privacy assessment.

Contribution

It presents the MIDST challenge, developing new black-box and white-box MIAs tailored for diffusion models on complex tabular data, advancing privacy evaluation methods.

Findings

Diffusion models show varying resistance to MIAs depending on data complexity.

Novel MIAs improve detection of privacy risks in synthetic tabular data.

The MIDST benchmark provides a standardized framework for privacy evaluation.

Abstract

Synthetic data is often perceived as a silver-bullet solution to data anonymization and privacy-preserving data publishing. Drawn from generative models like diffusion models, synthetic data is expected to preserve the statistical properties of the original dataset while remaining resilient to privacy attacks. Recent developments of diffusion models have been effective on a wide range of data types, but their privacy resilience, particularly for tabular formats, remains largely unexplored. MIDST challenge sought a quantitative evaluation of the privacy gain of synthetic tabular data generated by diffusion models, with a specific focus on its resistance to membership inference attacks (MIAs). Given the heterogeneity and complexity of tabular data, multiple target models were explored for MIAs, including diffusion models for single tables of mixed data types and multi-relational tables with interconnected constraints. MIDST inspired the development of novel black-box and white-box MIAs tailored to these target diffusion models as a key outcome, enabling a comprehensive evaluation of their privacy efficacy. The MIDST GitHub repository is available at https://github.com/VectorInstitute/MIDST