2D Instance Editing in 3D Space

TL;DR

This paper introduces a 2D-3D-2D framework for image editing that lifts 2D objects into 3D space for more consistent and realistic edits, then projects them back into 2D, outperforming existing methods.

Contribution

The novel framework enables physically plausible 3D-based editing of 2D images, improving consistency and object identity preservation over prior 2D-only approaches.

Findings

Outperforms existing 2D editing methods in consistency.

Preserves object identity more robustly.

Provides highly realistic and physically plausible edits.

Abstract

Generative models have achieved significant progress in advancing 2D image editing, demonstrating exceptional precision and realism. However, they often struggle with consistency and object identity preservation due to their inherent pixel-manipulation nature. To address this limitation, we introduce a novel "2D-3D-2D" framework. Our approach begins by lifting 2D objects into 3D representation, enabling edits within a physically plausible, rigidity-constrained 3D environment. The edited 3D objects are then reprojected and seamlessly inpainted back into the original 2D image. In contrast to existing 2D editing methods, such as DragGAN and DragDiffusion, our method directly manipulates objects in a 3D environment. Extensive experiments highlight that our framework surpasses previous methods in general performance, delivering highly consistent edits while robustly preserving object identity.