Lower Bounds on the MMSE of Adversarially Inferring Sensitive Features

TL;DR

This paper introduces a theoretical framework for evaluating the minimum mean-squared error (MMSE) in adversarial inference of sensitive features, providing bounds that account for finite samples and model restrictions, with practical empirical validation.

Contribution

It develops a novel adversarial evaluation framework with theoretical lower bounds on MMSE for sensitive feature inference, incorporating finite sample and model class errors, especially for linear models.

Findings

Derived closed-form bounds for linear models

Established order-optimal bounds related to noise variance

Validated the framework's effectiveness through empirical experiments

Abstract

We propose an adversarial evaluation framework for sensitive feature inference based on minimum mean-squared error (MMSE) estimation with a finite sample size and linear predictive models. Our approach establishes theoretical lower bounds on the true MMSE of inferring sensitive features from noisy observations of other correlated features. These bounds are expressed in terms of the empirical MMSE under a restricted hypothesis class and a non-negative error term. The error term captures both the estimation error due to finite number of samples and the approximation error from using a restricted hypothesis class. For linear predictive models, we derive closed-form bounds, which are order optimal in terms of the noise variance, on the approximation error for several classes of relationships between the sensitive and non-sensitive features, including linear mappings, binary symmetric channels, and class-conditional multi-variate Gaussian distributions. We also present a new lower bound that relies on the MSE computed on a hold-out validation dataset of the MMSE estimator learned on finite-samples and a restricted hypothesis class. Through empirical evaluation, we demonstrate that our framework serves as an effective tool for MMSE-based adversarial evaluation of sensitive feature inference that balances theoretical guarantees with practical efficiency.