Differentially Private Random Block Coordinate Descent

TL;DR

This paper introduces a differentially private random block coordinate descent method that generalizes existing private optimization algorithms, improves utility via importance sampling, and leverages coordinate-wise smoothness for better convergence.

Contribution

It proposes a novel DP coordinate descent algorithm that generalizes prior methods, incorporates importance sampling, and exploits coordinate heterogeneity for enhanced utility and convergence.

Findings

The new method generalizes DP-CD and DP-SGD.

Importance sampling improves utility.

Exploiting coordinate heterogeneity enhances convergence.

Abstract

Coordinate Descent (CD) methods have gained significant attention in machine learning due to their effectiveness in solving high-dimensional problems and their ability to decompose complex optimization tasks. However, classical CD methods were neither designed nor analyzed with data privacy in mind, a critical concern when handling sensitive information. This has led to the development of differentially private CD methods, such as DP-CD (Differentially Private Coordinate Descent) proposed by Mangold et al. (ICML 2022), yet a disparity remains between non-private CD and DP-CD methods. In our work, we propose a differentially private random block coordinate descent method that selects multiple coordinates with varying probabilities in each iteration using sketch matrices. Our algorithm generalizes both DP-CD and the classical DP-SGD (Differentially Private Stochastic Gradient Descent), while preserving the same utility guarantees. Furthermore, we demonstrate that better utility can be achieved through importance sampling, as our method takes advantage of the heterogeneity in coordinate-wise smoothness constants, leading to improved convergence rates.