Two-Party Privacy Games: How Users Perturb When Learners Preempt

TL;DR

This paper models the strategic interactions between users and learners in privacy-preserving machine learning, revealing conditions under which either users or learners perturb data or outputs, using game-theoretic analysis.

Contribution

It introduces a novel game-theoretic framework for analyzing user and learner perturbations in differential privacy, solving for equilibrium strategies in a two-party privacy game.

Findings

Either users or learners perturb data or outputs, but not both.

Learner perturbation occurs only when the number of users exceeds a certain threshold.

The threshold for learner perturbation increases with incentives misalignment.

Abstract

Internet tracking technologies and wearable electronics provide a vast amount of data to machine learning algorithms. This stock of data stands to increase with the developments of the internet of things and cyber-physical systems. Clearly, these technologies promise benefits. But they also raise the risk of sensitive information disclosure. To mitigate this risk, machine learning algorithms can add noise to outputs according to the formulations provided by differential privacy. At the same time, users can fight for privacy by injecting noise into the data that they report. In this paper, we conceptualize the interactions between privacy and accuracy and between user (input) perturbation and learner (output) perturbation in machine learning, using the frameworks of empirical risk minimization, differential privacy, and Stackelberg games. In particular, we solve for the Stackelberg equilibrium for the case of an averaging query. We find that, in equilibrium, either the users perturb their data before submission or the learner perturbs the machine learning output, but never both. Specifically, the learner perturbs if and only if the number of users is greater than a threshold which increases with the degree to which incentives are misaligned. Provoked by these conclusions - and by some observations from privacy ethics - we also suggest future directions. While other work in this area has studied privacy markets and mechanism design for truthful reporting of user information, we take a different viewpoint by considering both user and learner perturbation. We hope that this effort will open the door to future work in the area of differential privacy games.