A Modular Conditional Diffusion Framework for Image Reconstruction

TL;DR

This paper introduces a modular diffusion framework for image restoration that combines pre-trained models with minimal additional training, significantly reducing computation while maintaining high perceptual quality.

Contribution

It proposes a flexible, modular diffusion-based IR framework that requires training only a small task-specific module, enabling efficient adaptation across various IR tasks.

Findings

Outperforms existing methods in perceptual quality on multiple benchmarks.

Achieves at least four times reduction in neural function evaluations without performance loss.

Compatible with existing acceleration techniques like DDIM.

Abstract

Diffusion Probabilistic Models (DPMs) have been recently utilized to deal with various blind image restoration (IR) tasks, where they have demonstrated outstanding performance in terms of perceptual quality. However, the task-specific nature of existing solutions and the excessive computational costs related to their training, make such models impractical and challenging to use for different IR tasks than those that were initially trained for. This hinders their wider adoption, especially by those who lack access to powerful computational resources and vast amount of training data. In this work we aim to address the above issues and enable the successful adoption of DPMs in practical IR-related applications. Towards this goal, we propose a modular diffusion probabilistic IR framework (DP-IR), which allows us to combine the performance benefits of existing pre-trained state-of-the-art IR networks and generative DPMs, while it requires only the additional training of a relatively small module (0.7M params) related to the particular IR task of interest. Moreover, the architecture of the proposed framework allows for a sampling strategy that leads to at least four times reduction of neural function evaluations without suffering any performance loss, while it can also be combined with existing acceleration techniques such as DDIM. We evaluate our model on four benchmarks for the tasks of burst JDD-SR, dynamic scene deblurring, and super-resolution. Our method outperforms existing approaches in terms of perceptual quality while it retains a competitive performance with respect to fidelity metrics.