Can the accuracy bias by facial hairstyle be reduced through balancing the training data?

TL;DR

This study investigates whether balancing training data for facial hairstyles reduces accuracy bias in face recognition, finding that data balancing alone does not eliminate the accuracy gap caused by facial hair across demographics.

Contribution

The paper demonstrates that balancing training data by facial hairstyle does not significantly reduce recognition accuracy bias caused by facial hair.

Findings

Accuracy bias persists despite larger training sets.

Balanced datasets do not diminish accuracy variation.

Facial hair impacts recognition accuracy differently across demographics.

Abstract

Appearance of a face can be greatly altered by growing a beard and mustache. The facial hairstyles in a pair of images can cause marked changes to the impostor distribution and the genuine distribution. Also, different distributions of facial hairstyle across demographics could cause a false impression of relative accuracy across demographics. We first show that, even though larger training sets boost the recognition accuracy on all facial hairstyles, accuracy variations caused by facial hairstyles persist regardless of the size of the training set. Then, we analyze the impact of having different fractions of the training data represent facial hairstyles. We created balanced training sets using a set of identities available in Webface42M that both have clean-shaven and facial hair images. We find that, even when a face recognition model is trained with a balanced clean-shaven / facial hair training set, accuracy variation on the test data does not diminish. Next, data augmentation is employed to further investigate the effect of facial hair distribution in training data by manipulating facial hair pixels with the help of facial landmark points and a facial hair segmentation model. Our results show facial hair causes an accuracy gap between clean-shaven and facial hair images, and this impact can be significantly different between African-Americans and Caucasians.