GazeGaussian: High-Fidelity Gaze Redirection with 3D Gaussian Splatting

TL;DR

GazeGaussian introduces a high-fidelity gaze redirection method using a two-stream 3D Gaussian Splatting model that separately represents face and eye regions, enabling accurate gaze control and better generalization across subjects.

Contribution

It is the first to utilize a two-stream 3D Gaussian Splatting approach for high-fidelity gaze redirection with separate face and eye modeling and an expression-guided module for subject-specific synthesis.

Findings

Outperforms existing methods in rendering speed.

Achieves higher gaze redirection accuracy.

Enhances facial synthesis quality across datasets.

Abstract

Gaze estimation encounters generalization challenges when dealing with out-of-distribution data. To address this problem, recent methods use neural radiance fields (NeRF) to generate augmented data. However, existing methods based on NeRF are computationally expensive and lack facial details. 3D Gaussian Splatting (3DGS) has become the prevailing representation of neural fields. While 3DGS has been extensively examined in head avatars, it faces challenges with accurate gaze control and generalization across different subjects. In this work, we propose GazeGaussian, the first high-fidelity gaze redirection method that uses a two-stream 3DGS model to represent the face and eye regions separately. Leveraging the unstructured nature of 3DGS, we develop a novel representation of the eye for rigid eye rotation based on the target gaze direction. To enable synthesis generalization across various subjects, we integrate an expression-guided module to inject subject-specific information into the neural renderer. Comprehensive experiments show that GazeGaussian outperforms existing methods in rendering speed, gaze redirection accuracy, and facial synthesis across multiple datasets. The code is available at: https://ucwxb.github.io/GazeGaussian.