Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration

TL;DR

This paper introduces FPro, a frequency prompt guided transformer that decomposes image features into frequency components and uses dual prompt modulators to enhance various image restoration tasks, outperforming state-of-the-art methods.

Contribution

The novel Frequency Prompting method effectively incorporates frequency domain information into deep image restoration models, addressing limitations of spatial relation-based prompts.

Findings

FPro achieves superior performance on multiple image restoration benchmarks.

The frequency-based prompts improve artifact removal and detail preservation.

Experimental results demonstrate the effectiveness of dual prompt modulators.

Abstract

How to explore useful features from images as prompts to guide the deep image restoration models is an effective way to solve image restoration. In contrast to mining spatial relations within images as prompt, which leads to characteristics of different frequencies being neglected and further remaining subtle or undetectable artifacts in the restored image, we develop a Frequency Prompting image restoration method, dubbed FPro, which can effectively provide prompt components from a frequency perspective to guild the restoration model address these differences. Specifically, we first decompose input features into separate frequency parts via dynamically learned filters, where we introduce a gating mechanism for suppressing the less informative elements within the kernels. To propagate useful frequency information as prompt, we then propose a dual prompt block, consisting of a low-frequency prompt modulator (LPM) and a high-frequency prompt modulator (HPM), to handle signals from different bands respectively. Each modulator contains a generation process to incorporate prompting components into the extracted frequency maps, and a modulation part that modifies the prompt feature with the guidance of the decoder features. Experimental results on commonly used benchmarks have demonstrated the favorable performance of our pipeline against SOTA methods on 5 image restoration tasks, including deraining, deraindrop, demoir\'eing, deblurring, and dehazing. The source code and pre-trained models will be available at https://github.com/joshyZhou/FPro.