Calibrated Data-Dependent Constraints with Exact Satisfaction Guarantees

TL;DR

This paper introduces a calibration method for data-dependent constraints in machine learning, ensuring expected value constraints are satisfied with high probability, and demonstrates its effectiveness in fairness-sensitive classification.

Contribution

It proposes a novel calibration approach for data-dependent constraints that guarantees probabilistic satisfaction of expected constraints in machine learning models.

Findings

Effective in fairness-sensitive classification tasks

Guarantees probabilistic satisfaction of constraints

Compatible with standard stochastic optimization algorithms

Abstract

We consider the task of training machine learning models with data-dependent constraints. Such constraints often arise as empirical versions of expected value constraints that enforce fairness or stability goals. We reformulate data-dependent constraints so that they are calibrated: enforcing the reformulated constraints guarantees that their expected value counterparts are satisfied with a user-prescribed probability. The resulting optimization problem is amendable to standard stochastic optimization algorithms, and we demonstrate the efficacy of our method on a fairness-sensitive classification task where we wish to guarantee the classifier's fairness (at test time).