Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection

TL;DR

This paper introduces two efficient methods, Mean Projection and Tukey Median Projection, for removing specific information from embedding spaces with a single projection, reducing negative impacts compared to iterative nullspace projections.

Contribution

The paper proposes two novel single-projection methods, MP and TMP, that effectively remove targeted information while preserving other embedding information, improving over INLP.

Findings

MP removes linear separability of target information

MP has less impact on overall embedding space

Single MP projection is cleaner than multiple INLP projections

Abstract

Bias elimination and recent probing studies attempt to remove specific information from embedding spaces. Here it is important to remove as much of the target information as possible, while preserving any other information present. INLP is a popular recent method which removes specific information through iterative nullspace projections. Multiple iterations, however, increase the risk that information other than the target is negatively affected. We introduce two methods that find a single targeted projection: Mean Projection (MP, more efficient) and Tukey Median Projection (TMP, with theoretical guarantees). Our comparison between MP and INLP shows that (1) one MP projection removes linear separability based on the target and (2) MP has less impact on the overall space. Further analysis shows that applying random projections after MP leads to the same overall effects on the embedding space as the multiple projections of INLP. Applying one targeted (MP) projection hence is methodologically cleaner than applying multiple (INLP) projections that introduce random effects.