Pre-trained Encoders in Self-Supervised Learning Improve Secure and Privacy-preserving Supervised Learning

TL;DR

This paper systematically evaluates how pre-trained encoders in self-supervised learning enhance security and privacy in supervised learning, addressing limitations like accuracy loss and small security guarantees.

Contribution

It provides the first comprehensive measurement study demonstrating that pre-trained encoders improve security guarantees and accuracy in privacy-preserving supervised learning.

Findings

Pre-trained encoders improve accuracy under no attacks.

They enhance certified security against data poisoning and backdoor attacks.

They boost the effectiveness of differentially private classifiers.

Abstract

Classifiers in supervised learning have various security and privacy issues, e.g., 1) data poisoning attacks, backdoor attacks, and adversarial examples on the security side as well as 2) inference attacks and the right to be forgotten for the training data on the privacy side. Various secure and privacy-preserving supervised learning algorithms with formal guarantees have been proposed to address these issues. However, they suffer from various limitations such as accuracy loss, small certified security guarantees, and/or inefficiency. Self-supervised learning is an emerging technique to pre-train encoders using unlabeled data. Given a pre-trained encoder as a feature extractor, supervised learning can train a simple yet accurate classifier using a small amount of labeled training data. In this work, we perform the first systematic, principled measurement study to understand whether and when a pre-trained encoder can address the limitations of secure or privacy-preserving supervised learning algorithms. Our key findings are that a pre-trained encoder substantially improves 1) both accuracy under no attacks and certified security guarantees against data poisoning and backdoor attacks of state-of-the-art secure learning algorithms (i.e., bagging and KNN), 2) certified security guarantees of randomized smoothing against adversarial examples without sacrificing its accuracy under no attacks, 3) accuracy of differentially private classifiers, and 4) accuracy and/or efficiency of exact machine unlearning.