Contribution-based Low-Rank Adaptation with Pre-training Model for Real Image Restoration

TL;DR

This paper introduces CoLoRA, a contribution-based low-rank adaptation method for efficient fine-tuning of pre-trained models in image restoration, combined with PROD pre-training to enhance real-world performance and robustness.

Contribution

It proposes a novel contribution-based low-rank adaptation (CoLoRA) method for efficient fine-tuning in image restoration tasks, along with a new pre-training strategy (PROD) for improved robustness.

Findings

CoLoRA achieves performance comparable to full fine-tuning with fewer parameters.

PROD pre-training enhances robustness and bridges synthetic and real-world data.

The combined approach outperforms existing methods on diverse image restoration tasks.

Abstract

Recently, pre-trained model and efficient parameter tuning have achieved remarkable success in natural language processing and high-level computer vision with the aid of masked modeling and prompt tuning. In low-level computer vision, however, there have been limited investigations on pre-trained models and even efficient fine-tuning strategy has not yet been explored despite its importance and benefit in various real-world tasks such as alleviating memory inflation issue when integrating new tasks on AI edge devices. Here, we propose a novel efficient parameter tuning approach dubbed contribution-based low-rank adaptation (CoLoRA) for multiple image restorations along with effective pre-training method with random order degradations (PROD). Unlike prior arts that tune all network parameters, our CoLoRA effectively fine-tunes small amount of parameters by leveraging LoRA (low-rank adaptation) for each new vision task with our contribution-based method to adaptively determine layer by layer capacity for that task to yield comparable performance to full tuning. Furthermore, our PROD strategy allows to extend the capability of pre-trained models with improved performance as well as robustness to bridge synthetic pre-training and real-world fine-tuning. Our CoLoRA with PROD has demonstrated its superior performance in various image restoration tasks across diverse degradation types on both synthetic and real-world datasets for known and novel tasks.