Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries

TL;DR

This paper introduces a novel ensemble adversarial query method that significantly improves membership inference accuracy in machine learning models, especially in offline and low false-positive scenarios, by optimizing for discriminative and diverse queries.

Contribution

It proposes an adversarial approach to generate more effective queries for membership inference, surpassing existing techniques in accuracy and robustness.

Findings

Achieves higher inference accuracy than prior methods.

Excels in offline and low false-positive regimes.

Provides publicly available code for reproducibility.

Abstract

As industrial applications are increasingly automated by machine learning models, enforcing personal data ownership and intellectual property rights requires tracing training data back to their rightful owners. Membership inference algorithms approach this problem by using statistical techniques to discern whether a target sample was included in a model's training set. However, existing methods only utilize the unaltered target sample or simple augmentations of the target to compute statistics. Such a sparse sampling of the model's behavior carries little information, leading to poor inference capabilities. In this work, we use adversarial tools to directly optimize for queries that are discriminative and diverse. Our improvements achieve significantly more accurate membership inference than existing methods, especially in offline scenarios and in the low false-positive regime which is critical in legal settings. Code is available at https://github.com/YuxinWenRick/canary-in-a-coalmine.