PrivacyGuard: A Modular Framework for Privacy Auditing in Machine Learning

TL;DR

PrivacyGuard is a modular framework that empirically assesses privacy risks in machine learning models using diverse attacks and metrics, facilitating adaptable and comprehensive privacy auditing.

Contribution

It introduces a flexible, extensible tool for empirical differential privacy analysis in ML, integrating multiple attack methods and privacy metrics.

Findings

Supports various inference attacks like membership inference and reconstruction

Enables rapid adaptation with modular architecture

Provides a comprehensive suite for privacy risk evaluation

Abstract

The increasing deployment of Machine Learning (ML) models in sensitive domains motivates the need for robust, practical privacy assessment tools. PrivacyGuard is a comprehensive tool for empirical differential privacy (DP) analysis, designed to evaluate privacy risks in ML models through state-of-the-art inference attacks and advanced privacy measurement techniques. To this end, PrivacyGuard implements a diverse suite of privacy attack -- including membership inference , extraction, and reconstruction attacks -- enabling both off-the-shelf and highly configurable privacy analyses. Its modular architecture allows for the seamless integration of new attacks, and privacy metrics, supporting rapid adaptation to emerging research advances. We make PrivacyGuard available at https://github.com/facebookresearch/PrivacyGuard.