Distributionally-Robust Machine Learning Using Locally Differentially-Private Data

TL;DR

This paper introduces a distributionally-robust approach to machine learning with locally-differentially private data, using Wasserstein distance to define ambiguity sets and deriving new regularizers for regression models, including exact solutions for Gaussian data.

Contribution

It formulates privacy-preserving machine learning as a distributionally-robust optimization problem and derives novel regularizers, including an exact solution for Gaussian data, advancing privacy-aware model training.

Findings

New regularizer for linear regression models.

Exact solution for Gaussian data case.

Demonstrated improved performance on practical datasets.

Abstract

We consider machine learning, particularly regression, using locally-differentially private datasets. The Wasserstein distance is used to define an ambiguity set centered at the empirical distribution of the dataset corrupted by local differential privacy noise. The ambiguity set is shown to contain the probability distribution of unperturbed, clean data. The radius of the ambiguity set is a function of the privacy budget, spread of the data, and the size of the problem. Hence, machine learning with locally-differentially private datasets can be rewritten as a distributionally-robust optimization. For general distributions, the distributionally-robust optimization problem can relaxed as a regularized machine learning problem with the Lipschitz constant of the machine learning model as a regularizer. For linear and logistic regression, this regularizer is the dual norm of the model parameters. For Gaussian data, the distributionally-robust optimization problem can be solved exactly to find an optimal regularizer. This approach results in an entirely new regularizer for training linear regression models. Training with this novel regularizer can be posed as a semi-definite program. Finally, the performance of the proposed distributionally-robust machine learning training is demonstrated on practical datasets.