A hierarchical approach for assessing the vulnerability of tree-based classification models to membership inference attack

TL;DR

This paper introduces hierarchical, efficient methods for assessing the vulnerability of tree-based models to membership inference attacks, reducing the need for costly evaluations and enabling better privacy-preserving model tuning.

Contribution

It proposes ante-hoc hyperparameter analysis and post-hoc structural metrics as practical filters to identify high-risk models before expensive privacy attacks.

Findings

Hyperparameter risk prediction rules are highly accurate across datasets.

Model accuracy does not correlate with privacy risk.

Structural metrics effectively identify vulnerable models after training.

Abstract

Machine learning models can inadvertently expose confidential properties of their training data, making them vulnerable to membership inference attacks (MIA). While numerous evaluation methods exist, many require computationally expensive processes, such as training multiple shadow models. This article presents two new complementary approaches for efficiently identifying vulnerable tree-based models: an ante-hoc analysis of hyperparameter choices and a post-hoc examination of trained model structure. While these new methods cannot certify whether a model is safe from MIA, they provide practitioners with a means to significantly reduce the number of models that need to undergo expensive MIA assessment through a hierarchical filtering approach. More specifically, it is shown that the rank order of disclosure risk for different hyperparameter combinations remains consistent across datasets, enabling the development of simple, human-interpretable rules for identifying relatively high-risk models before training. While this ante-hoc analysis cannot determine absolute safety since this also depends on the specific dataset, it allows the elimination of unnecessarily risky configurations during hyperparameter tuning. Additionally, computationally inexpensive structural metrics serve as indicators of MIA vulnerability, providing a second filtering stage to identify risky models after training but before conducting expensive attacks. Empirical results show that hyperparameter-based risk prediction rules can achieve high accuracy in predicting the most at risk combinations of hyperparameters across different tree-based model types, while requiring no model training. Moreover, target model accuracy is not seen to correlate with privacy risk, suggesting opportunities to optimise model configurations for both performance and privacy.