Reference-Based 3D-Aware Image Editing with Triplanes

TL;DR

This paper introduces a novel framework for 3D-aware, reference-based image editing using triplane representations, enabling high-quality, diverse edits across multiple domains with state-of-the-art results.

Contribution

It presents an integrated approach combining encoding, localization, disentanglement, and fusion learning for effective 3D-aware image editing with triplanes.

Findings

Achieves high-quality edits across diverse domains.

Outperforms existing 2D and 3D-aware methods.

Demonstrates state-of-the-art quantitative and qualitative results.

Abstract

Generative Adversarial Networks (GANs) have emerged as powerful tools for high-quality image generation and real image editing by manipulating their latent spaces. Recent advancements in GANs include 3D-aware models such as EG3D, which feature efficient triplane-based architectures capable of reconstructing 3D geometry from single images. However, limited attention has been given to providing an integrated framework for 3D-aware, high-quality, reference-based image editing. This study addresses this gap by exploring and demonstrating the effectiveness of the triplane space for advanced reference-based edits. Our novel approach integrates encoding, automatic localization, spatial disentanglement of triplane features, and fusion learning to achieve the desired edits. We demonstrate how our approach excels across diverse domains, including human faces, 360-degree heads, animal faces, partially stylized edits like cartoon faces, full-body clothing edits, and edits on class-agnostic samples. Our method shows state-of-the-art performance over relevant latent direction, text, and image-guided 2D and 3D-aware diffusion and GAN methods, both qualitatively and quantitatively.