TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal

TL;DR

This paper introduces a parameter-efficient, task-aware prompt tuning framework for multi-task image restoration under adverse weather, achieving high performance with significantly fewer parameters.

Contribution

It proposes a novel two-stage training paradigm with enhanced, low-rank decomposed prompts that model inter-task relatedness, improving efficiency and accuracy in weather-related image restoration.

Findings

Achieves superior restoration performance with only 2.75M parameters.

Effectively models inter-task relatedness through contrastive constraints.

Enhances task-specific prompts with low-rank decomposition for better generalization.

Abstract

Image restoration under adverse weather conditions has been extensively explored, leading to numerous high-performance methods. In particular, recent advances in All-in-One approaches have shown impressive results by training on multi-task image restoration datasets. However, most of these methods rely on dedicated network modules or parameters for each specific degradation type, resulting in a significant parameter overhead. Moreover, the relatedness across different restoration tasks is often overlooked. In light of these issues, we propose a parameter-efficient All-in-One image restoration framework that leverages task-aware enhanced prompts to tackle various adverse weather degradations.Specifically, we adopt a two-stage training paradigm consisting of a pretraining phase and a prompt-tuning phase to mitigate parameter conflicts across tasks. We first employ supervised learning to acquire general restoration knowledge, and then adapt the model to handle specific degradation via trainable soft prompts. Crucially, we enhance these task-specific prompts in a task-aware manner. We apply low-rank decomposition to these prompts to capture both task-general and task-specific characteristics, and impose contrastive constraints to better align them with the actual inter-task relatedness. These enhanced prompts not only improve the parameter efficiency of the restoration model but also enable more accurate task modeling, as evidenced by t-SNE analysis. Experimental results on different restoration tasks demonstrate that the proposed method achieves superior performance with only 2.75M parameters.