Better Membership Inference Privacy Measurement through Discrepancy

TL;DR

This paper introduces a new scalable privacy metric based on discrepancy theory that better measures membership inference risks in large models, outperforming existing metrics and enabling practical privacy assessment.

Contribution

The authors propose a novel discrepancy-based privacy metric that scales to large models and provides a tighter upper bound on membership inference advantage, along with tailored attacks.

Findings

The new metric scales to large models like ImageNet.

It achieves higher advantage than existing metrics on recent models.

The approach does not require training multiple models.

Abstract

Membership Inference Attacks have emerged as a dominant method for empirically measuring privacy leakage from machine learning models. Here, privacy is measured by the {\em{advantage}} or gap between a score or a function computed on the training and the test data. A major barrier to the practical deployment of these attacks is that they do not scale to large well-generalized models -- either the advantage is relatively low, or the attack involves training multiple models which is highly compute-intensive. In this work, inspired by discrepancy theory, we propose a new empirical privacy metric that is an upper bound on the advantage of a family of membership inference attacks. We show that this metric does not involve training multiple models, can be applied to large Imagenet classification models in-the-wild, and has higher advantage than existing metrics on models trained with more recent and sophisticated training recipes. Motivated by our empirical results, we also propose new membership inference attacks tailored to these training losses.