A Note on the Significance Adjustment for FA*IR with Two Protected Groups

TL;DR

This paper improves the significance adjustment method in the FA*IR algorithm, enabling it to work effectively with very short rankings and low protected group proportions by precisely calculating minimum protected candidates and mapping significance levels.

Contribution

It introduces an exact calculation for minimum protected candidates and a binary search method to determine significance levels, enhancing FA*IR's applicability.

Findings

Exact calculation of minimum protected candidates per ranking position

Mapping from significance levels to discrete tables for better adjustment

Effective handling of short rankings with low protected group proportions

Abstract

In this report we provide an improvement of the significance adjustment from the FA*IR algorithm of Zehlike et al., which did not work for very short rankings in combination with a low minimum proportion $p$ for the protected group. We show how the minimum number of protected candidates per ranking position can be calculated exactly and provide a mapping from the continuous space of significance levels ($\alpha$) to a discrete space of tables, which allows us to find $\alpha_c$ using a binary search heuristic.