GeoDiffusion: A Training-Free Framework for Accurate 3D Geometric Conditioning in Image Generation

TL;DR

GeoDiffusion is a training-free framework that enables accurate 3D geometric control in image generation, improving efficiency and precision for design and creative applications by leveraging 3D priors and style transfer.

Contribution

It introduces GeoDiffusion, a novel training-free method for precise 3D geometric conditioning in image generation, with a new component GeoDrag for improved editing accuracy and speed.

Findings

Enables precise geometric modifications in image generation workflows.

Outperforms existing methods in accuracy and efficiency.

Demonstrates versatility across iterative design tasks.

Abstract

Precise geometric control in image generation is essential for engineering \& product design and creative industries to control 3D object features accurately in image space. Traditional 3D editing approaches are time-consuming and demand specialized skills, while current image-based generative methods lack accuracy in geometric conditioning. To address these challenges, we propose GeoDiffusion, a training-free framework for accurate and efficient geometric conditioning of 3D features in image generation. GeoDiffusion employs a class-specific 3D object as a geometric prior to define keypoints and parametric correlations in 3D space. We ensure viewpoint consistency through a rendered image of a reference 3D object, followed by style transfer to meet user-defined appearance specifications. At the core of our framework is GeoDrag, improving accuracy and speed of drag-based image editing on geometry guidance tasks and general instructions on DragBench. Our results demonstrate that GeoDiffusion enables precise geometric modifications across various iterative design workflows.