Hidden State Differential Private Mini-Batch Block Coordinate Descent for Multi-convexity Optimization

TL;DR

This paper introduces a new differentially private optimization algorithm for multi-convex problems, extending privacy guarantees to broader non-convex applications like neural networks and matrix factorization.

Contribution

The paper proposes the DP-MBCD algorithm with privacy loss accounting under the hidden state assumption, applicable to a wide range of non-convex problems.

Findings

Tighter privacy loss bounds achieved.

Applicable to non-convex problems like neural networks.

Compatible with proximal gradient descent and adaptive noise.

Abstract

We investigate the differential privacy (DP) guarantees under the hidden state assumption (HSA) for multi-convex problems. Recent analyses of privacy loss under the hidden state assumption have relied on strong assumptions such as convexity, thereby limiting their applicability to practical problems. In this paper, we introduce the Differential Privacy Mini-Batch Block Coordinate Descent (DP-MBCD) algorithm, accompanied by the privacy loss accounting methods under the hidden state assumption. Our proposed methods apply to a broad range of classical non-convex problems which are or can be converted to multi-convex problems, such as matrix factorization and neural network training. In addition to a tighter bound for privacy loss, our theoretical analysis is also compatible with proximal gradient descent and adaptive calibrated noise scenarios.