3D-Fixup: Advancing Photo Editing with 3D Priors
Yen-Chi Cheng, Krishna Kumar Singh, Jae Shin Yoon, Alex Schwing, Liangyan Gui, Matheus Gadelha, Paul Guerrero, Nanxuan Zhao

TL;DR
3D-Fixup introduces a framework that leverages learned 3D priors and diffusion models to enable realistic, identity-preserving 3D-aware edits of 2D images, including object translation and rotation.
Contribution
It presents a novel training-based approach that uses video data and 3D guidance from an Image-to-3D model to improve 3D-aware image editing capabilities.
Findings
Supports complex 3D transformations like translation and rotation
Achieves high-quality, identity-coherent edits
Effectively integrates 3D priors with diffusion models
Abstract
Despite significant advances in modeling image priors via diffusion models, 3D-aware image editing remains challenging, in part because the object is only specified via a single image. To tackle this challenge, we propose 3D-Fixup, a new framework for editing 2D images guided by learned 3D priors. The framework supports difficult editing situations such as object translation and 3D rotation. To achieve this, we leverage a training-based approach that harnesses the generative power of diffusion models. As video data naturally encodes real-world physical dynamics, we turn to video data for generating training data pairs, i.e., a source and a target frame. Rather than relying solely on a single trained model to infer transformations between source and target frames, we incorporate 3D guidance from an Image-to-3D model, which bridges this challenging task by explicitly projecting 2D…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsDiffusion
