Leakage of Dataset Properties in Multi-Party Machine Learning

TL;DR

This paper demonstrates that multi-party machine learning can unintentionally leak global dataset properties, such as attribute distributions, even when only black-box access to the final model is provided, raising privacy concerns.

Contribution

It reveals how dataset property leakage occurs in multi-party ML and analyzes factors influencing this leakage across various data types and models.

Findings

Leakage of population-level properties is possible even without direct attribute inclusion.

Leakage persists despite low correlation between sensitive attributes and other data.

Various data types, including tabular, text, and graph data, are vulnerable to this leakage.

Abstract

Secure multi-party machine learning allows several parties to build a model on their pooled data to increase utility while not explicitly sharing data with each other. We show that such multi-party computation can cause leakage of global dataset properties between the parties even when parties obtain only black-box access to the final model. In particular, a ``curious'' party can infer the distribution of sensitive attributes in other parties' data with high accuracy. This raises concerns regarding the confidentiality of properties pertaining to the whole dataset as opposed to individual data records. We show that our attack can leak population-level properties in datasets of different types, including tabular, text, and graph data. To understand and measure the source of leakage, we consider several models of correlation between a sensitive attribute and the rest of the data. Using multiple machine learning models, we show that leakage occurs even if the sensitive attribute is not included in the training data and has a low correlation with other attributes or the target variable.