A note on privacy preserving iteratively reweighted least squares

TL;DR

This paper introduces a practical privacy-preserving IRLS algorithm for linear models that overcomes sensitivity analysis and cumulative privacy loss challenges by leveraging concentrated differential privacy.

Contribution

It develops a novel privacy-preserving IRLS method that simplifies sensitivity analysis and reduces noise via concentrated differential privacy, improving accuracy.

Findings

The algorithm provides accurate private IRLS solutions.

Sensitivity analysis is simplified by treating matrix operations as post-processing.

Concentrated differential privacy reduces noise compared to traditional methods.

Abstract

Iteratively reweighted least squares (IRLS) is a widely-used method in machine learning to estimate the parameters in the generalised linear models. In particular, IRLS for L1 minimisation under the linear model provides a closed-form solution in each step, which is a simple multiplication between the inverse of the weighted second moment matrix and the weighted first moment vector. When dealing with privacy sensitive data, however, developing a privacy preserving IRLS algorithm faces two challenges. First, due to the inversion of the second moment matrix, the usual sensitivity analysis in differential privacy incorporating a single datapoint perturbation gets complicated and often requires unrealistic assumptions. Second, due to its iterative nature, a significant cumulative privacy loss occurs. However, adding a high level of noise to compensate for the privacy loss hinders from getting accurate estimates. Here, we develop a practical algorithm that overcomes these challenges and outputs privatised and accurate IRLS solutions. In our method, we analyse the sensitivity of each moments separately and treat the matrix inversion and multiplication as a post-processing step, which simplifies the sensitivity analysis. Furthermore, we apply the {\it{concentrated differential privacy}} formalism, a more relaxed version of differential privacy, which requires adding a significantly less amount of noise for the same level of privacy guarantee, compared to the conventional and advanced compositions of differentially private mechanisms.