MoE-DiffIR: Task-customized Diffusion Priors for Universal Compressed Image Restoration

TL;DR

MoE-DiffIR is a universal image restoration method that uses task-specific diffusion priors and a mixture-of-experts approach to adapt to various codecs and improve texture generation, especially at low bitrates.

Contribution

It introduces a task-customized diffusion prior framework with a mixture-of-experts prompt module and a visual-to-text adapter for enhanced adaptability and texture restoration in compressed images.

Findings

Demonstrates robustness across 21 degradation types and 7 codecs.

Achieves superior texture restoration, especially at low bitrates.

Outperforms existing methods in universal compressed image restoration.

Abstract

We present MoE-DiffIR, an innovative universal compressed image restoration (CIR) method with task-customized diffusion priors. This intends to handle two pivotal challenges in the existing CIR methods: (i) lacking adaptability and universality for different image codecs, e.g., JPEG and WebP; (ii) poor texture generation capability, particularly at low bitrates. Specifically, our MoE-DiffIR develops the powerful mixture-of-experts (MoE) prompt module, where some basic prompts cooperate to excavate the task-customized diffusion priors from Stable Diffusion (SD) for each compression task. Moreover, the degradation-aware routing mechanism is proposed to enable the flexible assignment of basic prompts. To activate and reuse the cross-modality generation prior of SD, we design the visual-to-text adapter for MoE-DiffIR, which aims to adapt the embedding of low-quality images from the visual domain to the textual domain as the textual guidance for SD, enabling more consistent and reasonable texture generation. We also construct one comprehensive benchmark dataset for universal CIR, covering 21 types of degradations from 7 popular traditional and learned codecs. Extensive experiments on universal CIR have demonstrated the excellent robustness and texture restoration capability of our proposed MoE-DiffIR. The project can be found at https://renyulin-f.github.io/MoE-DiffIR.github.io/.