Privacy Leakage via Output Label Space and Differentially Private Continual Learning

TL;DR

This paper uncovers a privacy side-channel in classification models' output label space, especially in continual learning, and proposes methods to mitigate it, improving privacy guarantees and model accuracy.

Contribution

It formalizes differential privacy for continual learning, identifies the label space as a privacy side-channel, and evaluates two effective mitigation strategies.

Findings

Models achieve higher accuracy under DP with proposed methods.

The label space side-channel is significant in continual learning.

Proposed methods effectively reduce privacy leakage.

Abstract

Differential privacy (DP) is a formal privacy framework that enables training machine learning (ML) models while protecting individuals' data. As pointed out by prior work, ML models are part of larger systems, which can lead to so-called privacy side-channels even if the model training itself is DP. We identify the output label space of a classification model as such a privacy side-channel and show a concrete privacy attack that exploits it. The side-channel becomes highly relevant in continual learning (CL), where the output label space changes over time. To reason about privacy guarantees in CL, we introduce a formalisation of DP for CL, which also clarifies how our approach differs from existing approaches. We propose and evaluate two methods for eliminating this side-channel: applying an optimal DP mechanism to release the labels in the sensitive data, and using a large public label space. We explore the trade-offs of these methods through adapting pre-trained models. We demonstrate empirically that our models consistently achieve higher accuracy under DP than previous work over both Split-CIFAR-100 and Split-ImageNet-R, with a stronger privacy model.