A Knowledge Transfer Framework for Differentially Private Sparse Learning

TL;DR

This paper introduces a knowledge transfer framework for differentially private high-dimensional sparse learning, improving utility guarantees in sparse regression tasks by distilling knowledge from private datasets.

Contribution

It proposes a novel framework that uses knowledge transfer from a teacher estimator to train a differentially private student estimator, with proven convergence and utility guarantees.

Findings

Achieves improved utility guarantees over previous methods

Demonstrates effectiveness on synthetic and real-world data

Provides theoretical analysis of convergence and privacy utility trade-offs

Abstract

We study the problem of estimating high dimensional models with underlying sparse structures while preserving the privacy of each training example. We develop a differentially private high-dimensional sparse learning framework using the idea of knowledge transfer. More specifically, we propose to distill the knowledge from a "teacher" estimator trained on a private dataset, by creating a new dataset from auxiliary features, and then train a differentially private "student" estimator using this new dataset. In addition, we establish the linear convergence rate as well as the utility guarantee for our proposed method. For sparse linear regression and sparse logistic regression, our method achieves improved utility guarantees compared with the best known results (Kifer et al., 2012; Wang and Gu, 2019). We further demonstrate the superiority of our framework through both synthetic and real-world data experiments.