Roll Your Eyes: Gaze Redirection via Explicit 3D Eyeball Rotation

TL;DR

This paper introduces a 3D gaze redirection method using explicit eyeball rotation with 3D Gaussian Splatting, outperforming neural radiance field-based approaches in image quality and gaze accuracy.

Contribution

It presents a novel explicit 3D eyeball structure for gaze redirection, incorporating 3D Gaussian Splatting and an adaptive deformation module for realistic muscle movement.

Findings

Achieves superior image quality over previous methods

Demonstrates accurate gaze estimation

Generates diverse, photorealistic gaze images

Abstract

We propose a novel 3D gaze redirection framework that leverages an explicit 3D eyeball structure. Existing gaze redirection methods are typically based on neural radiance fields, which employ implicit neural representations via volume rendering. Unlike these NeRF-based approaches, where the rotation and translation of 3D representations are not explicitly modeled, we introduce a dedicated 3D eyeball structure to represent the eyeballs with 3D Gaussian Splatting (3DGS). Our method generates photorealistic images that faithfully reproduce the desired gaze direction by explicitly rotating and translating the 3D eyeball structure. In addition, we propose an adaptive deformation module that enables the replication of subtle muscle movements around the eyes. Through experiments conducted on the ETH-XGaze dataset, we demonstrate that our framework is capable of generating diverse novel gaze images, achieving superior image quality and gaze estimation accuracy compared to previous state-of-the-art methods.