On the Inherent Privacy of Zeroth Order Projected Gradient Descent

TL;DR

This paper investigates whether the inherent randomness in zeroth-order gradient descent algorithms provides sufficient differential privacy, revealing that under certain conditions, it does not guarantee privacy and can lead to privacy loss growth.

Contribution

The work demonstrates that zeroth-order gradient descent can fail to ensure differential privacy, especially with fixed initialization or many iterations, challenging assumptions about inherent privacy.

Findings

Inherent noise in zeroth-order methods may not ensure privacy.

Privacy loss can grow superlinearly with iterations.

Fixed initialization can lead to non-private behavior.

Abstract

Differentially private zeroth-order optimization methods have recently gained popularity in private fine tuning of machine learning models due to their reduced memory requirements. Current approaches for privatizing zeroth-order methods rely on adding Gaussian noise to the estimated zeroth-order gradients. However, since the search direction in the zeroth-order methods is inherently random, researchers including Tang et al. (2024) and Zhang et al. (2024a) have raised an important question: is the inherent noise in zeroth-order estimators sufficient to ensure the overall differential privacy of the algorithm? This work settles this question for a class of oracle-based optimization algorithms where the oracle returns zeroth-order gradient estimates. In particular, we show that for a fixed initialization, there exist strongly convex objective functions such that running (Projected) Zeroth-Order Gradient Descent (ZO-GD) is not differentially private. Furthermore, we show that even with random initialization and without revealing (initial and) intermediate iterates, the privacy loss in ZO-GD can grow superlinearly with the number of iterations when minimizing convex objective functions.