Accuracy Improvement in Differentially Private Logistic Regression: A Pre-training Approach

TL;DR

This paper proposes a pre-training approach to enhance the accuracy of differentially private logistic regression models by pre-training on public data before fine-tuning on private data.

Contribution

It introduces a novel pre-training and fine-tuning framework specifically designed to improve DP logistic regression accuracy.

Findings

Pre-training significantly boosts DP-LR accuracy.

The method effectively balances privacy and utility.

Numerical results confirm the approach's effectiveness.

Abstract

Machine learning (ML) models can memorize training datasets. As a result, training ML models over private datasets can lead to the violation of individuals' privacy. Differential privacy (DP) is a rigorous privacy notion to preserve the privacy of underlying training datasets. Yet, training ML models in a DP framework usually degrades the accuracy of ML models. This paper aims to boost the accuracy of a DP logistic regression (LR) via a pre-training module. In more detail, we initially pre-train our LR model on a public training dataset that there is no privacy concern about it. Then, we fine-tune our DP-LR model with the private dataset. In the numerical results, we show that adding a pre-training module significantly improves the accuracy of the DP-LR model.