Beyond Membership: Limitations of Add/Remove Adjacency in Differential Privacy

TL;DR

This paper reveals that the common add/remove adjacency model in differential privacy overstates attribute privacy, demonstrating the importance of choosing the correct adjacency relation for accurate privacy guarantees.

Contribution

The authors identify limitations of add/remove adjacency in DP, develop novel attacks under substitute adjacency, and empirically show discrepancies in privacy guarantees.

Findings

Add/remove adjacency overstates attribute privacy compared to substitute adjacency.

Novel attacks demonstrate the gap between reported DP guarantees and actual privacy.

Empirical results show inconsistencies with add/remove-based privacy accounting.

Abstract

Training machine learning models with differential privacy (DP) limits an adversary's ability to infer sensitive information about the training data. It can be interpreted as a bound on adversary's capability to distinguish two adjacent datasets according to chosen adjacency relation. In practice, most DP implementations use the add/remove adjacency relation, where two datasets are adjacent if one can be obtained from the other by adding or removing a single record, thereby protecting membership. In many ML applications, however, the goal is to protect attributes of individual records (e.g., labels used in supervised fine-tuning). We show that privacy accounting under add/remove overstates attribute privacy compared to accounting under the substitute adjacency relation, which permits substituting one record. To demonstrate this gap, we develop novel attacks to audit DP under substitute adjacency, and show empirically that audit results are inconsistent with DP guarantees reported under add/remove, yet remain consistent with the budget accounted under the substitute adjacency relation. Our results highlight that the choice of adjacency when reporting DP guarantees is critical when the protection target is per-record attributes rather than membership.