Negative-Guided Subject Fidelity Optimization for Zero-Shot Subject-Driven Generation

TL;DR

This paper introduces Subject Fidelity Optimization (SFO), a new framework that improves zero-shot subject-driven image generation by using synthetic negatives and reweighted diffusion steps, leading to better subject detail preservation.

Contribution

The paper proposes SFO with Condition-Degradation Negative Sampling (CDNS) and timestep reweighting, advancing zero-shot subject-driven generation by explicitly guiding model focus on subject fidelity.

Findings

SFO with CDNS outperforms recent baselines in subject fidelity.

Reweighted diffusion timesteps enhance fine-grained subject detail learning.

Synthetic negatives improve the model's ability to distinguish subjects without human annotations.

Abstract

We present Subject Fidelity Optimization (SFO), a novel comparative learning framework for zero-shot subject-driven generation that enhances subject fidelity. Existing supervised fine-tuning methods, which rely only on positive targets and use the diffusion loss as in the pre-training stage, often fail to capture fine-grained subject details. To address this, SFO introduces additional synthetic negative targets and explicitly guides the model to favor positives over negatives through pairwise comparison. For negative targets, we propose Condition-Degradation Negative Sampling (CDNS), which automatically produces synthetic negatives tailored for subject-driven generation by introducing controlled degradations that emphasize subject fidelity and text alignment without expensive human annotations. Moreover, we reweight the diffusion timesteps to focus fine-tuning on intermediate steps where subject details emerge. Extensive experiments demonstrate that SFO with CDNS significantly outperforms recent strong baselines in terms of both subject fidelity and text alignment on a subject-driven generation benchmark. Project page: https://subjectfidelityoptimization.github.io/