Private Zeroth-Order Optimization with Public Data

TL;DR

This paper introduces PAZO, a public-data-assisted zeroth-order optimization framework that improves privacy-utility tradeoffs and computational efficiency in private machine learning tasks, especially in high privacy regimes.

Contribution

It proposes a novel PAZO framework leveraging public data to enhance private zeroth-order optimization, with theoretical analysis and empirical validation across vision and text tasks.

Findings

PAZO outperforms first-order baselines in privacy/utility tradeoffs.

PAZO achieves up to 16x runtime speedup.

Empirical results show superior performance in vision and text tasks.

Abstract

One of the major bottlenecks for deploying popular first-order differentially private (DP) machine learning algorithms (e.g., DP-SGD) lies in their high computation and memory cost, despite the existence of optimized implementations. Zeroth-order methods have promise in mitigating the overhead, as they leverage function evaluations to approximate the gradients, hence significantly easier to privatize. While recent works have explored zeroth-order approaches in both private and non-private settings, they still suffer from relatively low utilities compared with DP-SGD, and have only been evaluated in limited application domains. In this work, we propose to leverage public information to guide and improve gradient approximation of private zeroth-order algorithms. We explore a suite of public-data-assisted zeroth-order optimizers (PAZO) with minimal overhead. We provide theoretical analyses of the PAZO framework under an assumption of the similarity between public and private data. Empirically, we demonstrate that PAZO achieves superior privacy/utility tradeoffs across vision and text tasks in both pre-training and fine-tuning settings, outperforming the best first-order baselines (with public data) especially in highly private regimes, while offering up to $16\times$ runtime speedup.