Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference

TL;DR

This paper introduces Preference Score Distillation (PSD), a novel framework that uses pretrained 2D reward models and classifier-free guidance to improve human preference alignment in text-to-3D generation without requiring 3D training data.

Contribution

The paper proposes a new optimization-based method that leverages 2D reward models and CFG-style mechanisms for human-aligned text-to-3D synthesis, addressing data scarcity and performance constraints.

Findings

PSD outperforms existing methods in aesthetic metrics.

It seamlessly integrates with various pipelines.

The approach demonstrates strong extensibility.

Abstract

Human preference alignment presents a critical yet underexplored challenge for diffusion models in text-to-3D generation. Existing solutions typically require task-specific fine-tuning, posing significant hurdles in data-scarce 3D domains. To address this, we propose Preference Score Distillation (PSD), an optimization-based framework that leverages pretrained 2D reward models for human-aligned text-to-3D synthesis without 3D training data. Our key insight stems from the incompatibility of pixel-level gradients: due to the absence of noisy samples during reward model training, direct application of 2D reward gradients disturbs the denoising process. Noticing that similar issue occurs in the naive classifier guidance in conditioned diffusion models, we fundamentally rethink preference alignment as a classifier-free guidance (CFG)-style mechanism through our implicit reward model. Furthermore, recognizing that frozen pretrained diffusion models constrain performance, we introduce an adaptive strategy to co-optimize preference scores and negative text embeddings. By incorporating CFG during optimization, online refinement of negative text embeddings dynamically enhances alignment. To our knowledge, we are the first to bridge human preference alignment with CFG theory under score distillation framework. Experiments demonstrate the superiority of PSD in aesthetic metrics, seamless integration with diverse pipelines, and strong extensibility.