Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts

TL;DR

This paper introduces AdaptIR, a parameter-efficient Mixture-of-Experts approach that enhances generalization in image restoration across multiple degradation types by tuning only a small fraction of parameters.

Contribution

The paper proposes AdaptIR, a novel MoE-based framework with orthogonal multi-branch design for heterogeneous representations, improving generalization in image restoration with minimal parameter tuning.

Findings

Achieves stable performance on single-degradation tasks.

Excels in hybrid-degradation tasks.

Fine-tunes only 0.6% parameters in 8 hours.

Abstract

Designing single-task image restoration models for specific degradation has seen great success in recent years. To achieve generalized image restoration, all-in-one methods have recently been proposed and shown potential for multiple restoration tasks using one single model. Despite the promising results, the existing all-in-one paradigm still suffers from high computational costs as well as limited generalization on unseen degradations. In this work, we introduce an alternative solution to improve the generalization of image restoration models. Drawing inspiration from recent advancements in Parameter Efficient Transfer Learning (PETL), we aim to tune only a small number of parameters to adapt pre-trained restoration models to various tasks. However, current PETL methods fail to generalize across varied restoration tasks due to their homogeneous representation nature. To this end, we propose AdaptIR, a Mixture-of-Experts (MoE) with orthogonal multi-branch design to capture local spatial, global spatial, and channel representation bases, followed by adaptive base combination to obtain heterogeneous representation for different degradations. Extensive experiments demonstrate that our AdaptIR achieves stable performance on single-degradation tasks, and excels in hybrid-degradation tasks, with fine-tuning only 0.6% parameters for 8 hours.