Variational Latent Entropy Estimation Disentanglement: Controlled Attribute Leakage for Face Recognition

TL;DR

VLEED is a post-hoc disentanglement method that transforms face recognition embeddings to separate categorical attributes like gender and ethnicity, enhancing privacy and fairness.

Contribution

It introduces a variational autoencoder-based approach with mutual information objectives for stable training and fine-grained control over attribute disentanglement.

Findings

VLEED achieves better privacy-utility tradeoffs than existing methods.

It reduces demographic bias in face recognition.

The method maintains verification accuracy while disentangling attributes.

Abstract

Face recognition embeddings encode identity, but they also encode other factors such as gender and ethnicity. Depending on how these factors are used by a downstream system, separating them from the information needed for verification is important for both privacy and fairness. We propose Variational Latent Entropy Estimation Disentanglement (VLEED), a post-hoc method that transforms pretrained embeddings with a variational autoencoder and encourages a distilled representation where the categorical variable of interest is separated from identity-relevant information. VLEED uses a mutual information-based objective realised through the estimation of the entropy of the categorical attribute in the latent space, and provides stable training with fine-grained control over information removal. We evaluate our method on IJB-C, RFW, and VGGFace2 for gender and ethnicity disentanglement, and compare it to various state-of-the-art methods. We report verification utility, predictability of the disentangled variable under linear and nonlinear classifiers, and group disparity metrics based on false match rates. Our results show that VLEED offers a wide range of privacy-utility tradeoffs over existing methods and can also reduce recognition bias across demographic groups.