Evaluating the Impact of Local Differential Privacy on Utility Loss via Influence Functions

TL;DR

This paper introduces a method using influence functions to estimate how local differential privacy parameters impact model utility, enabling privacy-utility trade-offs without extensive retraining.

Contribution

The work demonstrates how influence functions can predict test loss changes due to local DP noise, reducing computational costs in privacy parameter tuning.

Findings

Influence functions accurately estimate test loss changes with small mean absolute error.

Method applies to binary and multi-class classification with various randomization scenarios.

Effective even when using label noise correction methods.

Abstract

How to properly set the privacy parameter in differential privacy (DP) has been an open question in DP research since it was first proposed in 2006. In this work, we demonstrate the ability of influence functions to offer insight into how a specific privacy parameter value will affect a model's test loss in the randomized response-based local DP setting. Our proposed method allows a data curator to select the privacy parameter best aligned with their allowed privacy-utility trade-off without requiring heavy computation such as extensive model retraining and data privatization. We consider multiple common randomization scenarios, such as performing randomized response over the features, and/or over the labels, as well as the more complex case of applying a class-dependent label noise correction method to offset the noise incurred by randomization. Further, we provide a detailed discussion over the computational complexity of our proposed approach inclusive of an empirical analysis. Through empirical evaluations we show that for both binary and multi-class settings, influence functions are able to approximate the true change in test loss that occurs when randomized response is applied over features and/or labels with small mean absolute error, especially in cases where noise correction methods are applied.