Distribution inference risks: Identifying and mitigating sources of leakage

TL;DR

This paper analyzes the causes of distribution inference leaks in machine learning models, identifying key sources of leakage and proposing mitigation strategies, including causal learning techniques, to enhance privacy protections.

Contribution

It provides a theoretical and empirical analysis of leakage sources in distribution inference attacks and introduces principled mitigation methods, notably causal learning, to improve model privacy.

Findings

Identified three main sources of leakage: memorization of $ ext{E}[Y|X]$, wrong inductive bias, and finite data.

Causal learning techniques are more resilient against distributional membership inference.

Formalized distribution inference to enable reasoning about more complex adversaries.

Abstract

A large body of work shows that machine learning (ML) models can leak sensitive or confidential information about their training data. Recently, leakage due to distribution inference (or property inference) attacks is gaining attention. In this attack, the goal of an adversary is to infer distributional information about the training data. So far, research on distribution inference has focused on demonstrating successful attacks, with little attention given to identifying the potential causes of the leakage and to proposing mitigations. To bridge this gap, as our main contribution, we theoretically and empirically analyze the sources of information leakage that allows an adversary to perpetrate distribution inference attacks. We identify three sources of leakage: (1) memorizing specific information about the $\mathbb{E}[Y|X]$ (expected label given the feature values) of interest to the adversary, (2) wrong inductive bias of the model, and (3) finiteness of the training data. Next, based on our analysis, we propose principled mitigation techniques against distribution inference attacks. Specifically, we demonstrate that causal learning techniques are more resilient to a particular type of distribution inference risk termed distributional membership inference than associative learning methods. And lastly, we present a formalization of distribution inference that allows for reasoning about more general adversaries than was previously possible.