Post-Processing in Local Differential Privacy: An Extensive Evaluation and Benchmark Platform

TL;DR

This paper provides an extensive evaluation and benchmark platform for post-processing methods in local differential privacy, analyzing their performance across various protocols, datasets, and utility metrics to guide practical application.

Contribution

It introduces LDP$^3$, a comprehensive, open-source benchmark platform for evaluating post-processing methods in local differential privacy, with extensive experimental analysis.

Findings

Post-processing improves utility under strict privacy budgets.

Optimal post-processing depends on multiple factors like protocol and data characteristics.

Benefit of post-processing diminishes as privacy budget increases.

Abstract

Local differential privacy (LDP) has recently gained prominence as a powerful paradigm for collecting and analyzing sensitive data from users' devices. However, the inherent perturbation added by LDP protocols reduces the utility of the collected data. To mitigate this issue, several post-processing (PP) methods have been developed. Yet, the comparative performance of PP methods under diverse settings remains underexplored. In this paper, we present an extensive benchmark comprising 6 popular LDP protocols, 7 PP methods, 4 utility metrics, and 6 datasets to evaluate the behaviors and optimality of PP methods under diverse conditions. Through extensive experiments, we show that while PP can substantially improve utility when the privacy budget is small (i.e., strict privacy), its benefit diminishes as the privacy budget grows. Moreover, our findings reveal that the optimal PP method depends on multiple factors, including the choice of LDP protocol, privacy budget, data characteristics (such as distribution and domain size), and the specific utility metric. To advance research in this area and assist practitioners in identifying the most suitable PP method for their setting, we introduce LDP$^3$, an open-source benchmark platform. LDP$^3$ contains all methods used in our experimental analysis, and it is designed in a modular, extensible, and multi-threaded way for future use and development.