From Same Photo: Cheating on Visual Kinship Challenges

TL;DR

This paper reveals that kinship verification models often rely on faces from the same photograph as an unintended signal, which can lead to misleadingly high accuracy, questioning the validity of current benchmarks.

Contribution

The study demonstrates that a simple classifier based on whether faces come from the same photo can achieve near state-of-the-art results on kinship verification datasets, exposing a data artifact.

Findings

Faces from the same photograph are a strong unintended signal.

Naive classifiers can outperform existing models using this artifact.

Current kinship datasets may overestimate true kinship signals.

Abstract

With the propensity for deep learning models to learn unintended signals from data sets there is always the possibility that the network can `cheat' in order to solve a task. In the instance of data sets for visual kinship verification, one such unintended signal could be that the faces are cropped from the same photograph, since faces from the same photograph are more likely to be from the same family. In this paper we investigate the influence of this artefactual data inference in published data sets for kinship verification. To this end, we obtain a large dataset, and train a CNN classifier to determine if two faces are from the same photograph or not. Using this classifier alone as a naive classifier of kinship, we demonstrate near state of the art results on five public benchmark data sets for kinship verification - achieving over 90% accuracy on one of them. Thus, we conclude that faces derived from the same photograph are a strong inadvertent signal in all the data sets we examined, and it is likely that the fraction of kinship explained by existing kinship models is small.