Training-Free Large Model Priors for Multiple-in-One Image Restoration

TL;DR

This paper introduces LMDIR, a training-free, multi-purpose image restoration framework that leverages large multi-modal language models and diffusion models to handle various degradations in a unified approach.

Contribution

The paper presents a novel multi-in-one image restoration method that integrates large model priors, enabling single-stage training for diverse degradation types without specialized models.

Findings

Outperforms state-of-the-art methods on multiple benchmarks.

Supports automatic and user-guided restoration.

Handles various degradations with a unified framework.

Abstract

Image restoration aims to reconstruct the latent clear images from their degraded versions. Despite the notable achievement, existing methods predominantly focus on handling specific degradation types and thus require specialized models, impeding real-world applications in dynamic degradation scenarios. To address this issue, we propose Large Model Driven Image Restoration framework (LMDIR), a novel multiple-in-one image restoration paradigm that leverages the generic priors from large multi-modal language models (MMLMs) and the pretrained diffusion models. In detail, LMDIR integrates three key prior knowledges: 1) global degradation knowledge from MMLMs, 2) scene-aware contextual descriptions generated by MMLMs, and 3) fine-grained high-quality reference images synthesized by diffusion models guided by MMLM descriptions. Standing on above priors, our architecture comprises a query-based prompt encoder, degradation-aware transformer block injecting global degradation knowledge, content-aware transformer block incorporating scene description, and reference-based transformer block incorporating fine-grained image priors. This design facilitates single-stage training paradigm to address various degradations while supporting both automatic and user-guided restoration. Extensive experiments demonstrate that our designed method outperforms state-of-the-art competitors on multiple evaluation benchmarks.