Second-Order Convergence in Private Stochastic Non-Convex Optimization

TL;DR

This paper introduces a new differentially private stochastic gradient descent framework that reliably finds second-order stationary points in non-convex optimization, improving convergence accuracy and utility in distributed settings.

Contribution

It proposes a generic PSGD framework using model drift for saddle point escape, eliminating the need for second-order info and private model selection, with extensions to distributed learning.

Findings

Achieves improved convergence error rates over prior methods.

Provides the first formal guarantees for DP-SOSP in distributed heterogeneous data.

Numerical experiments validate the effectiveness of the proposed approach.

Abstract

We investigate the problem of finding second-order stationary points (SOSP) in differentially private (DP) stochastic non-convex optimization. Existing methods suffer from two key limitations: (i) inaccurate convergence error rate due to overlooking gradient variance in the saddle point escape analysis, and (ii) dependence on auxiliary private model selection procedures for identifying DP-SOSP, which can significantly impair utility, particularly in distributed settings. To address these issues, we propose a generic perturbed stochastic gradient descent (PSGD) framework built upon Gaussian noise injection and general gradient oracles. A core innovation of our framework is using model drift distance to determine whether PSGD escapes saddle points, ensuring convergence to approximate local minima without relying on second-order information or additional DP-SOSP identification. By leveraging the adaptive DP-SPIDER estimator as a specific gradient oracle, we develop a new DP algorithm that rectifies the convergence error rates reported in prior work. We further extend this algorithm to distributed learning with heterogeneous data, providing the first formal guarantees for finding DP-SOSP in such settings. Our analysis also highlights the detrimental impacts of private selection procedures in distributed learning under high-dimensional models, underscoring the practical benefits of our design. Numerical experiments on real-world datasets validate the efficacy of our approach.