Your Privacy Depends on Others: Collusion Vulnerabilities in Individual Differential Privacy

TL;DR

This paper uncovers collusion vulnerabilities in individual differential privacy systems, showing that collective privacy choices can undermine individual guarantees and proposing a new privacy contract to address this issue.

Contribution

It reveals a hidden collusion risk in sampling-based iDP mechanisms and introduces a new privacy contract to provide users with explicit upper bounds on their excess vulnerability.

Findings

Empirical attacks succeeded against 62% of targeted individuals.

Colluding adversaries can amplify privacy risks within DP guarantees.

Proposed $(\varepsilon_i,\delta_i,\overline{ ext{Δ}})$-iDP offers a way to bound excess vulnerability.

Abstract

Individual Differential Privacy (iDP) promises users control over their privacy, but this promise can be broken in practice. We reveal a previously overlooked vulnerability in sampling-based iDP mechanisms: while conforming to the iDP guarantees, an individual's privacy risk is not solely governed by their own privacy budget, but critically depends on the privacy choices of all other data contributors. This creates a mismatch between the promise of individual privacy control and the reality of a system where risk is collectively determined. We demonstrate empirically that certain distributions of privacy preferences can unintentionally inflate the privacy risk of individuals, even when their formal guarantees are met. Moreover, this excess risk provides an exploitable attack vector. A central adversary or a set of colluding adversaries can deliberately choose privacy budgets to amplify vulnerabilities of targeted individuals. Most importantly, this attack operates entirely within the guarantees of DP, hiding this excess vulnerability. Our empirical evaluation demonstrates successful attacks against 62% of targeted individuals, substantially increasing their membership inference susceptibility. To mitigate this, we propose $(\varepsilon_i,\delta_i,\overline{\Delta})$-iDP a privacy contract that uses $\Delta$-divergences to provide users with a hard upper bound on their excess vulnerability, while offering flexibility to mechanism design. Our findings expose a fundamental challenge to the current paradigm, demanding a re-evaluation of how iDP systems are designed, audited, communicated, and deployed to make excess risks transparent and controllable.