Identity Overlap Between Face Recognition Train/Test Data: Causing Optimistic Bias in Accuracy Measurement

TL;DR

This paper investigates how identity overlap between training and test datasets in face recognition causes an overly optimistic accuracy estimate, emphasizing the need for identity-disjoint evaluation methods.

Contribution

It reveals the extent of identity overlap and label noise in common datasets, quantifies the bias introduced, and advocates for disjoint train/test splits in face recognition evaluation.

Findings

Significant identity overlap between MS1MV2 training set and LFW test sets.

Identity overlap causes an optimistic bias in accuracy estimates.

More challenging test sets exhibit larger bias.

Abstract

A fundamental tenet of pattern recognition is that overlap between training and testing sets causes an optimistic accuracy estimate. Deep CNNs for face recognition are trained for N-way classification of the identities in the training set. Accuracy is commonly estimated as average 10-fold classification accuracy on image pairs from test sets such as LFW, CALFW, CPLFW, CFP-FP and AgeDB-30. Because train and test sets have been independently assembled, images and identities in any given test set may also be present in any given training set. In particular, our experiments reveal a surprising degree of identity and image overlap between the LFW family of test sets and the MS1MV2 training set. Our experiments also reveal identity label noise in MS1MV2. We compare accuracy achieved with same-size MS1MV2 subsets that are identity-disjoint and not identity-disjoint with LFW, to reveal the size of the optimistic bias. Using more challenging test sets from the LFW family, we find that the size of the optimistic bias is larger for more challenging test sets. Our results highlight the lack of and the need for identity-disjoint train and test methodology in face recognition research.