PILLAR: How to make semi-private learning more effective

TL;DR

This paper introduces PILLAR, an efficient semi-private learning algorithm that leverages pre-trained features to reduce private labelled sample complexity and improve performance in low-data, privacy-constrained scenarios.

Contribution

The paper presents a novel algorithm that uses pre-trained network features to enhance semi-private learning, achieving lower sample complexity and better empirical results under strict privacy constraints.

Findings

Significantly lower private labelled sample complexity achieved.

Improved performance over baselines in low-data, high-privacy settings.

Effective use of pre-trained features from public data.

Abstract

In Semi-Supervised Semi-Private (SP) learning, the learner has access to both public unlabelled and private labelled data. We propose a computationally efficient algorithm that, under mild assumptions on the data, provably achieves significantly lower private labelled sample complexity and can be efficiently run on real-world datasets. For this purpose, we leverage the features extracted by networks pre-trained on public (labelled or unlabelled) data, whose distribution can significantly differ from the one on which SP learning is performed. To validate its empirical effectiveness, we propose a wide variety of experiments under tight privacy constraints ($\epsilon = 0.1$) and with a focus on low-data regimes. In all of these settings, our algorithm exhibits significantly improved performance over available baselines that use similar amounts of public data.