GeoDiffuser: Geometry-Based Image Editing with Diffusion Models
Rahul Sajnani, Jeroen Vanbaar, Jie Min, Kapil Katyal, Srinath Sridhar

TL;DR
GeoDiffuser is a zero-shot, geometry-based image editing method that unifies 2D and 3D edits within diffusion models by incorporating geometric transformations into attention layers, enabling precise, training-free edits.
Contribution
It introduces a novel, training-free approach that models image edits as geometric transformations integrated into diffusion models' attention layers, unifying 2D and 3D editing capabilities.
Findings
Outperforms existing methods in quantitative evaluations.
Effectively performs 2D and 3D edits like translation, rotation, and removal.
Achieves plausible edits with accurate lighting and shadows.
Abstract
The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However, these methods are bespoke, imprecise, require additional information, or are limited to only 2D image edits. We present GeoDiffuser, a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images, for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputer Graphics and Visualization Techniques · 3D Shape Modeling and Analysis · 3D Modeling in Geospatial Applications
MethodsSegment Anything Model · Diffusion
