Consistency and Accuracy of CelebA Attribute Values

TL;DR

This paper systematically analyzes the consistency and accuracy of facial attribute labels in CelebA, revealing significant inconsistencies and errors that impact model performance, and provides corrected attribute values for improved accuracy.

Contribution

It is the first comprehensive study exposing label inconsistencies in CelebA and offers corrected attribute data to enhance facial attribute classification accuracy.

Findings

Only 12 of 40 attributes have >= 95% labeling consistency.

High error rates found in manual audits, up to 40%.

Corrected attribute values improve model accuracy.

Abstract

We report the first systematic analysis of the experimental foundations of facial attribute classification. Two annotators independently assigning attribute values shows that only 12 of 40 common attributes are assigned values with >= 95% consistency, and three (high cheekbones, pointed nose, oval face) have essentially random consistency. Of 5,068 duplicate face appearances in CelebA, attributes have contradicting values on from 10 to 860 of the 5,068 duplicates. Manual audit of a subset of CelebA estimates error rates as high as 40% for (no beard=false), even though the labeling consistency experiment indicates that no beard could be assigned with >= 95% consistency. Selecting the mouth slightly open (MSO) for deeper analysis, we estimate the error rate for (MSO=true) at about 20% and (MSO=false) at about 2%. A corrected version of the MSO attribute values enables learning a model that achieves higher accuracy than previously reported for MSO. Corrected values for CelebA MSO are available at https://github.com/HaiyuWu/CelebAMSO.