Privacy Side Channels in Machine Learning Systems

TL;DR

This paper reveals how system-level components in machine learning systems can be exploited as privacy side channels, enabling attacks like membership inference and data exfiltration that undermine privacy guarantees.

Contribution

It introduces the concept of privacy side channels in ML systems, categorizes them across the ML lifecycle, and demonstrates their potential to breach privacy protections.

Findings

Deduplicating training data can invalidate differential privacy guarantees.

Blocking language model regeneration can be exploited to exfiltrate private keys.

System-level components can be exploited to perform privacy attacks beyond standalone models.

Abstract

Most current approaches for protecting privacy in machine learning (ML) assume that models exist in a vacuum. Yet, in reality, these models are part of larger systems that include components for training data filtering, output monitoring, and more. In this work, we introduce privacy side channels: attacks that exploit these system-level components to extract private information at far higher rates than is otherwise possible for standalone models. We propose four categories of side channels that span the entire ML lifecycle (training data filtering, input preprocessing, output post-processing, and query filtering) and allow for enhanced membership inference, data extraction, and even novel threats such as extraction of users' test queries. For example, we show that deduplicating training data before applying differentially-private training creates a side-channel that completely invalidates any provable privacy guarantees. We further show that systems which block language models from regenerating training data can be exploited to exfiltrate private keys contained in the training set--even if the model did not memorize these keys. Taken together, our results demonstrate the need for a holistic, end-to-end privacy analysis of machine learning systems.