GazeProphetV2: Head-Movement-Based Gaze Prediction Enabling Efficient Foveated Rendering on Mobile VR

TL;DR

GazeProphetV2 introduces a multimodal gaze prediction method combining gaze history, head movement, and scene data with attention mechanisms, significantly improving accuracy for VR rendering and interaction without expensive eye tracking.

Contribution

The paper presents a novel multimodal approach with cross-modal attention for gaze prediction in VR, enhancing accuracy and generalization over previous single-modality methods.

Findings

Improved gaze prediction accuracy across multiple VR scenes.

High temporal consistency in predicted gaze trajectories.

Effective generalization to unseen scenes with 93.1% validation accuracy.

Abstract

Predicting gaze behavior in virtual reality environments remains a significant challenge with implications for rendering optimization and interface design. This paper introduces a multimodal approach to VR gaze prediction that combines temporal gaze patterns, head movement data, and visual scene information. By leveraging a gated fusion mechanism with cross-modal attention, the approach learns to adaptively weight gaze history, head movement, and scene content based on contextual relevance. Evaluations using a dataset spanning 22 VR scenes with 5.3M gaze samples demonstrate improvements in predictive accuracy when combining modalities compared to using individual data streams alone. The results indicate that integrating past gaze trajectories with head orientation and scene content enhances prediction accuracy across 1-3 future frames. Cross-scene generalization testing shows consistent performance with 93.1% validation accuracy and temporal consistency in predicted gaze trajectories. These findings contribute to understanding attention mechanisms in virtual environments while suggesting potential applications in rendering optimization, interaction design, and user experience evaluation. The approach represents a step toward more efficient virtual reality systems that can anticipate user attention patterns without requiring expensive eye tracking hardware.