GO-Renderer: Generative Object Rendering with 3D-aware Controllable Video Diffusion Models

TL;DR

GO-Renderer is a unified framework that combines 3D reconstruction and diffusion models to enable high-quality, controllable object rendering from arbitrary viewpoints and lighting conditions, surpassing previous methods.

Contribution

It introduces a novel integration of 3D proxies with diffusion models for accurate viewpoint and lighting control in object rendering.

Findings

Achieves state-of-the-art results in viewpoint synthesis

Enables rendering in novel lighting environments

Supports object insertion into existing videos

Abstract

Reconstructing a renderable 3D model from images is a useful but challenging task. Recent feedforward 3D reconstruction methods have demonstrated remarkable success in efficiently recovering geometry, but still cannot accurately model the complex appearances of these 3D reconstructed models. Recent diffusion-based generative models can synthesize realistic images or videos of an object using reference images without explicitly modeling its appearance, which provides a promising direction for object rendering, but lacks accurate control over the viewpoints. In this paper, we propose GO-Renderer, a unified framework integrating the reconstructed 3D proxies to guide the video generative models to achieve high-quality object rendering on arbitrary viewpoints under arbitrary lighting conditions. Our method not only enjoys the accurate viewpoint control using the reconstructed 3D proxy but also enables high-quality rendering in different lighting environments using diffusion generative models without explicitly modeling complex materials and lighting. Extensive experiments demonstrate that GO-Renderer achieves state-of-the-art performance across the object rendering tasks, including synthesizing images on new viewpoints, rendering the objects in a novel lighting environment, and inserting an object into an existing video.