On the Importance of Difficulty Calibration in Membership Inference Attacks

TL;DR

This paper demonstrates that calibrating the difficulty of samples in membership inference attacks greatly reduces false positives, making such attacks more reliable without sacrificing accuracy.

Contribution

It introduces difficulty calibration as a novel method to improve the reliability of membership inference attacks by reducing false positives.

Findings

Difficulty calibration significantly lowers false positive rates.

Calibrated attacks maintain high accuracy.

Improved attack reliability in practical scenarios.

Abstract

The vulnerability of machine learning models to membership inference attacks has received much attention in recent years. However, existing attacks mostly remain impractical due to having high false positive rates, where non-member samples are often erroneously predicted as members. This type of error makes the predicted membership signal unreliable, especially since most samples are non-members in real world applications. In this work, we argue that membership inference attacks can benefit drastically from \emph{difficulty calibration}, where an attack's predicted membership score is adjusted to the difficulty of correctly classifying the target sample. We show that difficulty calibration can significantly reduce the false positive rate of a variety of existing attacks without a loss in accuracy.