On Theoretical Limits of Learning with Label Differential Privacy

TL;DR

This paper explores the fundamental theoretical limits of learning with label differential privacy, revealing how relaxing privacy constraints to labels alone can significantly improve learning rates under local DP, but offers limited gains under central DP.

Contribution

It establishes minimax convergence rate bounds for label DP in classification and regression, and compares the benefits of local versus central DP relaxations.

Findings

Under label local DP, the convergence rate is significantly faster than full DP.

Relaxing to label DP under local DP offers substantial performance improvements.

Under label central DP, the risk reduction is only a constant factor compared to full DP.

Abstract

Label differential privacy (DP) is designed for learning problems involving private labels and public features. While various methods have been proposed for learning under label DP, the theoretical limits remain largely unexplored. In this paper, we investigate the fundamental limits of learning with label DP in both local and central models for both classification and regression tasks, characterized by minimax convergence rates. We establish lower bounds by converting each task into a multiple hypothesis testing problem and bounding the test error. Additionally, we develop algorithms that yield matching upper bounds. Our results demonstrate that under label local DP (LDP), the risk has a significantly faster convergence rate than that under full LDP, i.e. protecting both features and labels, indicating the advantages of relaxing the DP definition to focus solely on labels. In contrast, under the label central DP (CDP), the risk is only reduced by a constant factor compared to full DP, indicating that the relaxation of CDP only has limited benefits on the performance.