Active Fourier Auditor for Estimating Distributional Properties of ML Models

TL;DR

This paper introduces the Active Fourier Auditor (AFA), a novel framework that efficiently estimates ML model properties like robustness and fairness by leveraging Fourier coefficients without reconstructing the model.

Contribution

The paper proposes a new Fourier-based auditing framework and the AFA method, which improves accuracy and sample efficiency over existing approaches.

Findings

AFA provides more accurate property estimates than baselines.

AFA reduces sample complexity in property estimation.

Theoretical error bounds support AFA's effectiveness.

Abstract

With the pervasive deployment of Machine Learning (ML) models in real-world applications, verifying and auditing properties of ML models have become a central concern. In this work, we focus on three properties: robustness, individual fairness, and group fairness. We discuss two approaches for auditing ML model properties: estimation with and without reconstruction of the target model under audit. Though the first approach is studied in the literature, the second approach remains unexplored. For this purpose, we develop a new framework that quantifies different properties in terms of the Fourier coefficients of the ML model under audit but does not parametrically reconstruct it. We propose the Active Fourier Auditor (AFA), which queries sample points according to the Fourier coefficients of the ML model, and further estimates the properties. We derive high probability error bounds on AFA's estimates, along with the worst-case lower bounds on the sample complexity to audit them. Numerically we demonstrate on multiple datasets and models that AFA is more accurate and sample-efficient to estimate the properties of interest than the baselines.