Quadratic Objective Perturbation: Curvature-Based Differential Privacy

TL;DR

This paper introduces Quadratic Objective Perturbation (QOP), a novel differential privacy method that uses curvature-based quadratic perturbations to relax gradient boundedness assumptions and improve privacy guarantees.

Contribution

QOP extends objective perturbation by perturbing with a quadratic form, enabling privacy under weaker assumptions and providing theoretical and numerical advantages over existing methods.

Findings

QOP achieves $(\,\varepsilon, \delta)$-differential privacy under weaker conditions.

QOP maintains privacy guarantees even with approximate solutions.

QOP offers improved utility and efficiency compared to Linear Objective Perturbation (LOP).

Abstract

Objective perturbation is a standard mechanism in differentially private empirical risk minimization. In particular, Linear Objective Perturbation (LOP) enforces privacy by adding a random linear term, while strong convexity and stability are ensured by an additional deterministic quadratic term. However, this approach requires the strong assumption of bounded gradients of the loss function, which excludes many modern machine learning models. In this work, we introduce Quadratic Objective Perturbation (QOP), which perturbs the objective with a random quadratic form. This perturbation induces strong convexity and enforces stability of the problem through curvature, thereby enabling privacy and allowing sensitivity to be controlled through spectral properties of the perturbation rather than assumptions on the gradients. As a result, we obtain $(\varepsilon, \delta)$-differential privacy under weaker assumptions, in the interpolation regime. Furthermore, we extend the analysis to account for approximate solutions, showing that privacy guarantees are preserved under inexact solves. Additionally, we derive utility guarantees in terms of empirical excess risk, and provide a theoretical and numerical comparison to LOP, highlighting the advantages of curvature-based perturbations. Finally, we discuss algorithmic aspects and show that the resulting problems can be solved efficiently using modern splitting schemes.