The importance of feature preprocessing for differentially private linear optimization

TL;DR

This paper demonstrates that feature preprocessing is essential for effective differentially private linear classification, showing that DPSGD alone may not suffice and proposing an improved algorithm with theoretical guarantees and practical validation.

Contribution

The paper reveals the importance of feature preprocessing in differentially private linear optimization and introduces DPSGD-F, an algorithm combining DPSGD with preprocessing, with proven optimality bounds.

Findings

Without preprocessing, DPSGD can have large optimality gaps.

DPSGD-F achieves optimality gap proportional to feature diameter.

Practical validation on image classification benchmarks.

Abstract

Training machine learning models with differential privacy (DP) has received increasing interest in recent years. One of the most popular algorithms for training differentially private models is differentially private stochastic gradient descent (DPSGD) and its variants, where at each step gradients are clipped and combined with some noise. Given the increasing usage of DPSGD, we ask the question: is DPSGD alone sufficient to find a good minimizer for every dataset under privacy constraints? Towards answering this question, we show that even for the simple case of linear classification, unlike non-private optimization, (private) feature preprocessing is vital for differentially private optimization. In detail, we first show theoretically that there exists an example where without feature preprocessing, DPSGD incurs an optimality gap proportional to the maximum Euclidean norm of features over all samples. We then propose an algorithm called DPSGD-F, which combines DPSGD with feature preprocessing and prove that for classification tasks, it incurs an optimality gap proportional to the diameter of the features $\max_{x, x' \in D} \|x - x'\|_2$. We finally demonstrate the practicality of our algorithm on image classification benchmarks.