ScanGAN360: A Generative Model of Realistic Scanpaths for 360$^{\circ}$ Images

TL;DR

ScanGAN360 is a novel generative adversarial network designed to produce realistic scanpaths for 360-degree images, improving over existing methods and closely mimicking human gaze behavior for virtual environment analysis.

Contribution

We introduce ScanGAN360, a specialized GAN that models realistic 360-degree scanpaths using spherical adaptations and a new loss function, advancing gaze prediction in immersive environments.

Findings

Outperforms existing methods significantly

Scanpaths are nearly on par with human gaze patterns

Enables fast virtual observer simulation

Abstract

Understanding and modeling the dynamics of human gaze behavior in 360$^\circ$ environments is a key challenge in computer vision and virtual reality. Generative adversarial approaches could alleviate this challenge by generating a large number of possible scanpaths for unseen images. Existing methods for scanpath generation, however, do not adequately predict realistic scanpaths for 360$^\circ$ images. We present ScanGAN360, a new generative adversarial approach to address this challenging problem. Our network generator is tailored to the specifics of 360$^\circ$ images representing immersive environments. Specifically, we accomplish this by leveraging the use of a spherical adaptation of dynamic-time warping as a loss function and proposing a novel parameterization of 360$^\circ$ scanpaths. The quality of our scanpaths outperforms competing approaches by a large margin and is almost on par with the human baseline. ScanGAN360 thus allows fast simulation of large numbers of virtual observers, whose behavior mimics real users, enabling a better understanding of gaze behavior and novel applications in virtual scene design.