Neural Photofit: Gaze-based Mental Image Reconstruction

TL;DR

This paper introduces a neural network-based approach that uses gaze data to reconstruct facial images from mental images, combining encoding, scoring, and decoding networks trained on gaze data.

Contribution

It presents a novel neural framework that decodes mental images into photofits using gaze data, outperforming baseline methods and validated by human studies.

Findings

Significantly outperforms baseline predictors

Produces visually plausible and accurate photofits

Validated by human subject study

Abstract

We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and neural attention and predicts a relevance score for each extracted image feature. Finally, image features are aggregated into a single feature vector as a linear combination of all features weighted by relevance which a decoder decodes into the final photofit. We train the neural scoring network on a novel dataset containing gaze data of 19 participants looking at collages of synthetic faces. We show that our method significantly outperforms a mean baseline predictor and report on a human study that shows that we can decode photofits that are visually plausible and close to the observer's mental image.