OSLO: One-Shot Label-Only Membership Inference Attacks

TL;DR

OSLO is a novel one-shot label-only membership inference attack that accurately determines if a sample was in a model's training set using only a single query, outperforming existing methods significantly.

Contribution

The paper introduces OSLO, a new attack leveraging transfer-based adversarial attacks to perform high-precision membership inference with just one query, unlike previous methods requiring thousands.

Findings

OSLO achieves at least 7× higher TPR at 1% FPR compared to previous attacks.

OSLO requires only a single query, reducing the number of queries by orders of magnitude.

OSLO outperforms state-of-the-art label-only MIAs in precision and true positive rate.

Abstract

We introduce One-Shot Label-Only (OSLO) membership inference attacks (MIAs), which accurately infer a given sample's membership in a target model's training set with high precision using just \emph{a single query}, where the target model only returns the predicted hard label. This is in contrast to state-of-the-art label-only attacks which require $\sim6000$ queries, yet get attack precisions lower than OSLO's. OSLO leverages transfer-based black-box adversarial attacks. The core idea is that a member sample exhibits more resistance to adversarial perturbations than a non-member. We compare OSLO against state-of-the-art label-only attacks and demonstrate that, despite requiring only one query, our method significantly outperforms previous attacks in terms of precision and true positive rate (TPR) under the same false positive rates (FPR). For example, compared to previous label-only MIAs, OSLO achieves a TPR that is at least 7$\times$ higher under a 1\% FPR and at least 22$\times$ higher under a 0.1\% FPR on CIFAR100 for a ResNet18 model. We evaluated multiple defense mechanisms against OSLO.