Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards

TL;DR

Nabla-R2D3 is a reinforcement learning framework that effectively aligns 3D diffusion models with 2D rewards, improving realism and instruction-following in 3D asset generation with high efficiency.

Contribution

It introduces a novel reinforcement learning method based on Nabla-GFlowNet for efficient 3D diffusion model alignment using 2D rewards, overcoming prior limitations.

Findings

Achieves higher rewards compared to baselines.

Converges quickly with fewer finetuning steps.

Reduces prior forgetting during alignment.

Abstract

Generating high-quality and photorealistic 3D assets remains a longstanding challenge in 3D vision and computer graphics. Although state-of-the-art generative models, such as diffusion models, have made significant progress in 3D generation, they often fall short of human-designed content due to limited ability to follow instructions, align with human preferences, or produce realistic textures, geometries, and physical attributes. In this paper, we introduce Nabla-R2D3, a highly effective and sample-efficient reinforcement learning alignment framework for 3D-native diffusion models using 2D rewards. Built upon the recently proposed Nabla-GFlowNet method, which matches the score function to reward gradients in a principled manner for reward finetuning, our Nabla-R2D3 enables effective adaptation of 3D diffusion models using only 2D reward signals. Extensive experiments show that, unlike vanilla finetuning baselines which either struggle to converge or suffer from reward hacking, Nabla-R2D3 consistently achieves higher rewards and reduced prior forgetting within a few finetuning steps.