Boundary-targeted Membership Inference Attacks on Safety Classifiers

TL;DR

This paper introduces a boundary-targeted membership inference attack on safety classifiers in generative AI, revealing privacy vulnerabilities and evaluating mitigation strategies.

Contribution

It proposes a novel boundary-targeted selection strategy for MIAs and demonstrates its effectiveness over existing methods in privacy leakage scenarios.

Findings

Adversaries can recover 19% of flagged conversations with 5% false positives.

The new attack is 3.5 times more effective than state-of-the-art MIAs.

Content filtering is ineffective; noise strategies can mitigate susceptibility.

Abstract

Safety classifiers are essential safeguards within generative AI systems, filtering harmful content or identifying at-risk users when interacting with large language models. Despite their necessity, these models are trained on sensitive datasets including discussions of self-harm and mental health, raising important, yet poorly understood, privacy concerns. Membership inference attacks (MIAs) allow adversaries to infer membership of examples used to train models. In this work, we hypothesize that identifying the examples on which the classifier is least confident are informative for an adversary to infer membership. This reflects a localized failure of generalization, where the model relies on memorization to resolve ambiguity in the training set. To investigate this, we introduce a new boundary-targeted selection strategy that identifies low confidence examples that amplify the signal of an examples membership within a training set. Our experimental results show that an adversary can recover 19\% of the conversations a safety classifier flagged as indicating user distress, at a 5\% false-positive rate, on a classifier fine-tuned for detecting a user who may require emotional support. This is $3.5$ times more than attacking using state-of-the-art MIA methods alone. Finally, we characterize the boundary laying examples and show that content-based filtering is ineffective for protection, and existing noise strategies can effectively mitigate susceptibility of these examples.