Coarse-Guided Visual Generation via Weighted h-Transform Sampling

TL;DR

This paper introduces a novel sampling-guided visual generation method using the h-transform, enabling high-quality, guided synthesis from coarse references without extensive training.

Contribution

It proposes a training-free, h-transform-based approach that effectively balances guidance and quality in visual generation tasks.

Findings

Effective guidance in diverse image and video generation tasks.

Outperforms existing training-free methods in quality and control.

Generalizes well across different types of visual data.

Abstract

Coarse-guided visual generation, which synthesizes fine visual samples from degraded or low-fidelity coarse references, is essential for various real-world applications. While training-based approaches are effective, they are inherently limited by high training costs and restricted generalization due to paired data collection. Accordingly, recent training-free works propose to leverage pretrained diffusion models and incorporate guidance during the sampling process. However, these training-free methods either require knowing the forward (fine-to-coarse) transformation operator, e.g., bicubic downsampling, or are difficult to balance between guidance and synthetic quality. To address these challenges, we propose a novel guided method by using the h-transform, a tool that can constrain stochastic processes (e.g., sampling process) under desired conditions. Specifically, we modify the transition probability at each sampling timestep by adding to the original differential equation with a drift function, which approximately steers the generation toward the ideal fine sample. To address unavoidable approximation errors, we introduce a noise-level-aware schedule that gradually de-weights the term as the error increases, ensuring both guidance adherence and high-quality synthesis. Extensive experiments across diverse image and video generation tasks demonstrate the effectiveness and generalization of our method.