Score Distillation Beyond Acceleration: Generative Modeling from Corrupted Data
Yasi Zhang, Tianyu Chen, Zhendong Wang, Ying Nian Wu, Mingyuan Zhou, Oscar Leong
TL;DR
This paper introduces Restoration Score Distillation (RSD), a framework for training high-quality generative models directly from corrupted data, achieving faster sampling and improved quality without needing clean images.
Contribution
RSD unifies corrupted data modeling with score distillation, enabling efficient one-step generative models from degraded observations across various corruption types.
Findings
RSD reduces FID scores across multiple datasets and tasks.
It achieves up to 30x faster sampling than multi-step diffusion models.
RSD outperforms diffusion teachers without access to clean data.
Abstract
Learning generative models directly from corrupted observations is a long standing challenge across natural and scientific domains. We introduce Restoration Score Distillation (RSD), a unified framework for learning high fidelity, one step generative models using only degraded data and the mapping may be the identity or a non invertible corruption operator (e.g., blur, masking, subsampling, Fourier acquisition). RSD first pretrains a corruption aware diffusion teacher on the observed measurements, then distills it into an efficient one step generator whose samples are statistically closer to the clean distribution p_X. The framework subsumes identity corruption (denoising task) as a special case of our general formulation. Empirically, RSD consistently reduces Frechet Inception Distance (FID) relative to corruption aware diffusion teachers across noisy generation (CIFAR 10, FFHQ,…
Peer Reviews
Decision·ICLR 2026 Poster
1. The paper introduces DCD, a novel framework for learning generative models from corrupted data without needing clean data. This approach is original in its unified treatment of diverse corruptions. 2. The research is of high quality, with a rigorous two-phase framework combining corruption-aware diffusion pretraining and score distillation. Theoretical analysis supports the method, and comprehensive experiments validate its effectiveness across various datasets and tasks. 3. The proposed DC
1. The DCD framework extends score distillation to corrupted data but lacks novel methodological contributions. Score distillation has already been widely used for learning generative models from clean data, and is extended to the scenario of corrupted data in this paper. However, it seems that DCD relies on existing diffusion models and score distillation techniques and offers limited innovations and does not provide unique paradigm for learning from corrupted data. 2. In the distillation pha
1. The paper addresses an important practical problem. Learning from corrupted data is particularly useful in scientific areas like medical imaging. A notable contribution of this paper is that it shows that score distillation can improve generation quality from corrupted data in addition to accelerating sampling. 2. The method is empirically strong. The proposed method is tested on multiple datasets, diverse corruption types, and a real-world application. The gains over the teacher models are
1. The primary novelty of framework lies in specific composition of established techniques and the empirical validation that distillation is particularly effective in this setting. It leverages corruption-aware diffusion training and score distillation as constituent elements but is limited in inventing new components. The manuscript would be strengthened by more precise positioning of its contribution and clarifies its novelty and contribution in synthesizing existing methods to solve a new pro
- **Originality:** Combines corruption-aware diffusion training with score distillation in a unified framework, with a focus on real-world scenarios where clean data is inaccessible. - **Quality:** Extensive experiments across multiple tasks, datasets, and noise levels demonstrate robust and consistent improvements. - **Clarity:** The paper is well-written, with clear explanations of both methodology and experimental design. - **Significance:** Addresses a critical challenge in generative modeli
- **Limited Comparison to GANs:** While diffusion-based methods are the focus, a comparison with GAN-based approaches trained on corrupted data could provide a broader perspective. - **Complexity of Theoretical Analysis:** The theoretical section is dense and may be challenging for readers without a strong background in divergence metrics and Gaussian analysis. - **Hyperparameter Sensitivity:** The performance of distillation losses (e.g., SiD vs. DMD) may depend on hyperparameter tuning, which
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Model Reduction and Neural Networks · Advanced Neuroimaging Techniques and Applications
MethodsDiffusion