FineFACE: Fair Facial Attribute Classification Leveraging Fine-grained Features

TL;DR

This paper introduces FineFACE, a novel fine-grained classification approach that improves fairness and accuracy in facial attribute classification without needing demographic annotations, by leveraging multi-level features and attention mechanisms.

Contribution

It presents a new method framing fair facial attribute classification as a fine-grained problem, integrating local and semantic features via cross-layer attention, achieving Pareto-efficient fairness and accuracy.

Findings

Improves accuracy by 1.32% to 1.74% over SOTA.

Enhances fairness by 67% to 83.6%.

Achieves Pareto-efficient fairness-accuracy trade-off.

Abstract

Published research highlights the presence of demographic bias in automated facial attribute classification algorithms, particularly impacting women and individuals with darker skin tones. Existing bias mitigation techniques typically require demographic annotations and often obtain a trade-off between fairness and accuracy, i.e., Pareto inefficiency. Facial attributes, whether common ones like gender or others such as "chubby" or "high cheekbones", exhibit high interclass similarity and intraclass variation across demographics leading to unequal accuracy. This requires the use of local and subtle cues using fine-grained analysis for differentiation. This paper proposes a novel approach to fair facial attribute classification by framing it as a fine-grained classification problem. Our approach effectively integrates both low-level local features (like edges and color) and high-level semantic features (like shapes and structures) through cross-layer mutual attention learning. Here, shallow to deep CNN layers function as experts, offering category predictions and attention regions. An exhaustive evaluation on facial attribute annotated datasets demonstrates that our FineFACE model improves accuracy by 1.32% to 1.74% and fairness by 67% to 83.6%, over the SOTA bias mitigation techniques. Importantly, our approach obtains a Pareto-efficient balance between accuracy and fairness between demographic groups. In addition, our approach does not require demographic annotations and is applicable to diverse downstream classification tasks. To facilitate reproducibility, the code and dataset information is available at https://github.com/VCBSL-Fairness/FineFACE.