WAYLA - Generating Images from Eye Movements

TL;DR

This paper introduces WAYLA, a system that reconstructs images viewed by observers solely from eye movement data using Conditional GANs, demonstrating effective image synthesis for newspapers and text images.

Contribution

The paper presents a novel eye-tracking based image reconstruction method utilizing Conditional GANs, specifically for newspapers and text images, advancing gaze-based image synthesis.

Findings

Successfully reconstructs newspaper images from gaze heat maps

Achieves detailed image reconstruction of text-containing images

Validated with quantitative metrics and human inspection

Abstract

We present a method for reconstructing images viewed by observers based only on their eye movements. By exploring the relationships between gaze patterns and image stimuli, the "What Are You Looking At?" (WAYLA) system learns to synthesize photo-realistic images that are similar to the original pictures being viewed. The WAYLA approach is based on the Conditional Generative Adversarial Network (Conditional GAN) image-to-image translation technique of Isola et al. We consider two specific applications - the first, of reconstructing newspaper images from gaze heat maps, and the second, of detailed reconstruction of images containing only text. The newspaper image reconstruction process is divided into two image-to-image translation operations, the first mapping gaze heat maps into image segmentations, and the second mapping the generated segmentation into a newspaper image. We validate the performance of our approach using various evaluation metrics, along with human visual inspection. All results confirm the ability of our network to perform image generation tasks using eye tracking data.