3D Object Manipulation in a Single Image using Generative Models

TL;DR

OMG3D is a novel framework that combines geometric control with diffusion models to enable realistic 3D object manipulation and motion editing in images, improving visual fidelity in static and dynamic scenarios.

Contribution

The paper introduces OMG3D, integrating 3D conversion, texture refinement, and lighting correction for improved image manipulation using diffusion models.

Findings

Enhanced visual realism in object editing and motion

Effective texture and lighting refinement modules

Operable with a single NVIDIA 3090 GPU

Abstract

Object manipulation in images aims to not only edit the object's presentation but also gift objects with motion. Previous methods encountered challenges in concurrently handling static editing and dynamic generation, while also struggling to achieve fidelity in object appearance and scene lighting. In this work, we introduce \textbf{OMG3D}, a novel framework that integrates the precise geometric control with the generative power of diffusion models, thus achieving significant enhancements in visual performance. Our framework first converts 2D objects into 3D, enabling user-directed modifications and lifelike motions at the geometric level. To address texture realism, we propose CustomRefiner, a texture refinement module that pre-train a customized diffusion model, aligning the details and style of coarse renderings of 3D rough model with the original image, further refine the texture. Additionally, we introduce IllumiCombiner, a lighting processing module that estimates and corrects background lighting to match human visual perception, resulting in more realistic shadow effects. Extensive experiments demonstrate the outstanding visual performance of our approach in both static and dynamic scenarios. Remarkably, all these steps can be done using one NVIDIA 3090. Project page is at https://whalesong-zrs.github.io/OMG3D-projectpage/