Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

TL;DR

This paper introduces Score Jacobian Chaining, a method that leverages pretrained 2D diffusion models for 3D generation by applying the chain rule to learned gradients and using a differentiable renderer, addressing distribution mismatch issues.

Contribution

The paper presents a novel technique to adapt 2D diffusion models for 3D data generation through Jacobian chaining and a new estimation mechanism for distribution mismatch.

Findings

Successfully applied to off-the-shelf diffusion models including Stable Diffusion.

Achieved 3D generation by aggregating multi-view 2D scores.

Resolved distribution mismatch with a novel estimation method.

Abstract

A diffusion model learns to predict a vector field of gradients. We propose to apply chain rule on the learned gradients, and back-propagate the score of a diffusion model through the Jacobian of a differentiable renderer, which we instantiate to be a voxel radiance field. This setup aggregates 2D scores at multiple camera viewpoints into a 3D score, and repurposes a pretrained 2D model for 3D data generation. We identify a technical challenge of distribution mismatch that arises in this application, and propose a novel estimation mechanism to resolve it. We run our algorithm on several off-the-shelf diffusion image generative models, including the recently released Stable Diffusion trained on the large-scale LAION dataset.