Goldilocks Test Sets for Face Verification

TL;DR

This paper introduces challenging, high-quality face verification test sets that reveal weaknesses in current models by focusing on attribute variation and similar-looking identities, avoiding artificial image degradation.

Contribution

The authors propose three new face verification test sets addressing attribute differences and similar-looking persons, with rigorous assembly rules for fair evaluation.

Findings

Test sets are more challenging than existing ones without image quality reduction.

Models show significant performance drops on the new test sets.

Proposed datasets reveal overlooked weaknesses in face verification algorithms.

Abstract

Reported face verification accuracy has reached a plateau on current well-known test sets. As a result, some difficult test sets have been assembled by reducing the image quality or adding artifacts to the image. However, we argue that test sets can be challenging without artificially reducing the image quality because the face recognition (FR) models suffer from correctly recognizing 1) the pairs from the same identity (i.e., genuine pairs) with a large face attribute difference, 2) the pairs from different identities (i.e., impostor pairs) with a small face attribute difference, and 3) the pairs of similar-looking identities (e.g., twins and relatives). We propose three challenging test sets to reveal important but ignored weaknesses of the existing FR algorithms. To challenge models on variation of facial attributes, we propose Hadrian and Eclipse to address facial hair differences and face exposure differences. The images in both test sets are high-quality and collected in a controlled environment. To challenge FR models on similar-looking persons, we propose ND-Twins, which contains images from a dedicated twins dataset. The LFW test protocol is used to structure the proposed test sets. Moreover, we introduce additional rules to assemble ``Goldilocks\footnote{https://en.wikipedia.org/wiki/Goldilocks_and_the_Three_Bears}" level test sets, including 1) restricted number of occurrence of hard samples, 2) equal chance evaluation across demographic groups, and 3) constrained identity overlap across validation folds. Quantitatively, without further processing the images, the proposed test sets have on-par or higher difficulties than the existing test sets that add artifacts to the images. The datasets are available at: https://github.com/HaiyuWu/SOTA-Face-Recognition-Train-and-Test.