RTGaze: Real-Time 3D-Aware Gaze Redirection from a Single Image

TL;DR

RTGaze is a real-time, 3D-aware gaze redirection method that produces high-quality, controllable face images from a single image, significantly outperforming previous methods in speed and accuracy.

Contribution

This work introduces RTGaze, a novel neural rendering approach that achieves real-time, high-quality gaze redirection with 3D consistency from a single face image.

Findings

RTGaze operates at approximately 0.06 seconds per image.

It outperforms previous methods in redirection accuracy and image quality.

RTGaze is 800 times faster than prior 3D-aware gaze redirection techniques.

Abstract

Gaze redirection methods aim to generate realistic human face images with controllable eye movement. However, recent methods often struggle with 3D consistency, efficiency, or quality, limiting their practical applications. In this work, we propose RTGaze, a real-time and high-quality gaze redirection method. Our approach learns a gaze-controllable facial representation from face images and gaze prompts, then decodes this representation via neural rendering for gaze redirection. Additionally, we distill face geometric priors from a pretrained 3D portrait generator to enhance generation quality. We evaluate RTGaze both qualitatively and quantitatively, demonstrating state-of-the-art performance in efficiency, redirection accuracy, and image quality across multiple datasets. Our system achieves real-time, 3D-aware gaze redirection with a feedforward network (~0.06 sec/image), making it 800x faster than the previous state-of-the-art 3D-aware methods.