DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360{\deg} Images

TL;DR

DiffGaze introduces a diffusion model that generates realistic, diverse continuous gaze sequences on 360-degree images, advancing human gaze prediction and animation with superior performance and naturalness.

Contribution

It presents a novel diffusion-based approach using transformers for continuous gaze sequence generation on 360-degree images, outperforming existing methods.

Findings

Outperforms state-of-the-art on benchmark datasets

Generates gaze sequences indistinguishable from real humans in user study

Effective for gaze prediction, scanpath, and saliency tasks

Abstract

We present DiffGaze, a novel method for generating realistic and diverse continuous human gaze sequences on 360{\deg} images based on a conditional score-based denoising diffusion model. Generating human gaze on 360{\deg} images is important for various human-computer interaction and computer graphics applications, e.g. for creating large-scale eye tracking datasets or for realistic animation of virtual humans. However, existing methods are limited to predicting discrete fixation sequences or aggregated saliency maps, thereby neglecting crucial parts of natural gaze behaviour. Our method uses features extracted from 360{\deg} images as condition and uses two transformers to model the temporal and spatial dependencies of continuous human gaze. We evaluate DiffGaze on two 360{\deg} image benchmarks for gaze sequence generation as well as scanpath prediction and saliency prediction. Our evaluations show that DiffGaze outperforms state-of-the-art methods on all tasks on both benchmarks. We also report a 21-participant user study showing that our method generates gaze sequences that are indistinguishable from real human sequences.