Global Modeling Matters: A Fast, Lightweight and Effective Baseline for Efficient Image Restoration

TL;DR

This paper introduces PW-FNet, a novel, efficient image restoration baseline that leverages pyramid wavelet and Fourier transforms to achieve high-quality results with reduced complexity, suitable for real-time applications.

Contribution

The paper proposes PW-FNet, a lightweight, multi-scale, multi-frequency image restoration network using wavelet and Fourier transforms, addressing the limitations of transformer-based methods.

Findings

Outperforms state-of-the-art methods in various restoration tasks.

Reduces computational complexity and inference time.

Maintains high restoration quality with fewer parameters.

Abstract

Natural image quality is often degraded by adverse weather conditions, significantly impairing the performance of downstream tasks. Image restoration has emerged as a core solution to this challenge and has been widely discussed in the literature. Although recent transformer-based approaches have made remarkable progress in image restoration, their increasing system complexity poses significant challenges for real-time processing, particularly in real-world deployment scenarios. To this end, most existing methods attempt to simplify the self-attention mechanism, such as by channel self-attention or state space model. However, these methods primarily focus on network architecture while neglecting the inherent characteristics of image restoration itself. In this context, we explore a pyramid Wavelet-Fourier iterative pipeline to demonstrate the potential of Wavelet-Fourier processing for image restoration. Inspired by the above findings, we propose a novel and efficient restoration baseline, named Pyramid Wavelet-Fourier Network (PW-FNet). Specifically, PW-FNet features two key design principles: 1) at the inter-block level, integrates a pyramid wavelet-based multi-input multi-output structure to achieve multi-scale and multi-frequency bands decomposition; and 2) at the intra-block level, incorporates Fourier transforms as an efficient alternative to self-attention mechanisms, effectively reducing computational complexity while preserving global modeling capability. Extensive experiments on tasks such as image deraining, raindrop removal, image super-resolution, motion deblurring, image dehazing, image desnowing and underwater/low-light enhancement demonstrate that PW-FNet not only surpasses state-of-the-art methods in restoration quality but also achieves superior efficiency, with significantly reduced parameter size, computational cost and inference time.