GeoDiffuser: Geometry-Based Image Editing with Diffusion Models

TL;DR

GeoDiffuser is a zero-shot, geometry-based image editing method that unifies 2D and 3D edits within diffusion models by incorporating geometric transformations into attention layers, enabling precise, training-free edits.

Contribution

It introduces a novel, training-free approach that models image edits as geometric transformations integrated into diffusion models' attention layers, unifying 2D and 3D editing capabilities.

Findings

Outperforms existing methods in quantitative evaluations.

Effectively performs 2D and 3D edits like translation, rotation, and removal.

Achieves plausible edits with accurate lighting and shadows.

Abstract

The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However, these methods are bespoke, imprecise, require additional information, or are limited to only 2D image edits. We present GeoDiffuser, a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images, for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input, we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation, 3D rotation, and removal. We present quantitative results, including a perceptual study, that shows how our approach is better than existing methods. Visit https://ivl.cs.brown.edu/research/geodiffuser.html for more information.