Human face perception reflects inverse-generative and naturalistic discriminative objectives

TL;DR

This study compares neural network models trained on different face perception tasks to human judgments, revealing that models focusing on invariant, high-level features best match human perception.

Contribution

It demonstrates that models trained on natural images and emphasizing invariant facial features most closely replicate human face recognition.

Findings

Models trained on natural images outperform synthetic-trained models.

High-level, invariant structure-focused models best match human judgments.

Natural image training enhances model alignment with human face perception.

Abstract

The perceptual representations supporting our ability to recognize faces remain a computational mystery. Deep neural networks offer mechanistic hypotheses for human face perception, but theoretically distinct models often make indistinguishable representational predictions for randomly sampled faces. To expose diagnostic differences among these hypotheses, we compared six neural network models sharing an architecture but trained on distinct tasks, using face pairs optimized to elicit contrasting model predictions ("controversial" pairs) alongside randomly sampled pairs. We tested model predictions against face-dissimilarity judgments from 864 human participants across stimulus sets differing in realism and pose variation. Models prioritizing high-level, invariant structures (trained via inverse rendering, face identification, or object classification) most robustly matched human judgments. Furthermore, models trained on natural images typically outperformed synthetic-trained counterparts. Together, these findings suggest that human face perception is shaped by mechanisms that infer latent causes of facial appearance, discount nuisance variation, and are tuned by natural image statistics.