Maximum Weighted Loss Discrepancy

TL;DR

This paper introduces maximum weighted loss discrepancy (MWLD), a measure of inequality in loss across groups, and explores its properties, estimation methods, and impact on fairness and robustness in machine learning models.

Contribution

The paper defines MWLD, analyzes its properties, proposes efficient estimation methods for certain weights, and demonstrates its practical benefits in fairness and robustness.

Findings

MWLD is related to group fairness and robustness.

Efficient estimation of MWLD is possible for specific weights.

Loss variance regularization reduces MWLD and maintains accuracy.

Abstract

Though machine learning algorithms excel at minimizing the average loss over a population, this might lead to large discrepancies between the losses across groups within the population. To capture this inequality, we introduce and study a notion we call maximum weighted loss discrepancy (MWLD), the maximum (weighted) difference between the loss of a group and the loss of the population. We relate MWLD to group fairness notions and robustness to demographic shifts. We then show MWLD satisfies the following three properties: 1) It is statistically impossible to estimate MWLD when all groups have equal weights. 2) For a particular family of weighting functions, we can estimate MWLD efficiently. 3) MWLD is related to loss variance, a quantity that arises in generalization bounds. We estimate MWLD with different weighting functions on four common datasets from the fairness literature. We finally show that loss variance regularization can halve the loss variance of a classifier and hence reduce MWLD without suffering a significant drop in accuracy.