Face Verification with Challenging Imposters and Diversified Demographics

TL;DR

This paper introduces the FaVCI2D dataset to address limitations in existing face verification datasets by including challenging imposter pairs and diverse demographics, revealing performance gaps in state-of-the-art models.

Contribution

The creation of the FaVCI2D dataset with challenging imposter pairs and demographic metadata, highlighting the need for more robust evaluation in face verification.

Findings

State-of-the-art models perform significantly worse on FaVCI2D.

Existing datasets are too simplistic and lack demographic diversity.

Performance drops confirm the need for more challenging benchmarks.

Abstract

Face verification aims to distinguish between genuine and imposter pairs of faces, which include the same or different identities, respectively. The performance reported in recent years gives the impression that the task is practically solved. Here, we revisit the problem and argue that existing evaluation datasets were built using two oversimplifying design choices. First, the usual identity selection to form imposter pairs is not challenging enough because, in practice, verification is needed to detect challenging imposters. Second, the underlying demographics of existing datasets are often insufficient to account for the wide diversity of facial characteristics of people from across the world. To mitigate these limitations, we introduce the $FaVCI2D$ dataset. Imposter pairs are challenging because they include visually similar faces selected from a large pool of demographically diversified identities. The dataset also includes metadata related to gender, country and age to facilitate fine-grained analysis of results. $FaVCI2D$ is generated from freely distributable resources. Experiments with state-of-the-art deep models that provide nearly 100\% performance on existing datasets show a significant performance drop for $FaVCI2D$, confirming our starting hypothesis. Equally important, we analyze legal and ethical challenges which appeared in recent years and hindered the development of face analysis research. We introduce a series of design choices which address these challenges and make the dataset constitution and usage more sustainable and fairer. $FaVCI2D$ is available at~\url{https://github.com/AIMultimediaLab/FaVCI2D-Face-Verification-with-Challenging-Imposters-and-Diversified-Demographics}.