Deep Fourier Up-Sampling

TL;DR

This paper introduces Deep Fourier Up-Sampling, a novel method leveraging Fourier domain properties for global modeling in neural networks, improving multi-scale feature up-sampling across various vision tasks.

Contribution

It presents a theoretically grounded Fourier-based up-sampling operator that can be integrated into existing networks, addressing limitations of local pixel attention methods.

Findings

Consistent performance improvements across multiple vision tasks.

Effective global modeling through Fourier domain transformations.

Enhanced up-sampling quality compared to traditional spatial methods.

Abstract

Existing convolutional neural networks widely adopt spatial down-/up-sampling for multi-scale modeling. However, spatial up-sampling operators (\emph{e.g.}, interpolation, transposed convolution, and un-pooling) heavily depend on local pixel attention, incapably exploring the global dependency. In contrast, the Fourier domain obeys the nature of global modeling according to the spectral convolution theorem. Unlike the spatial domain that performs up-sampling with the property of local similarity, up-sampling in the Fourier domain is more challenging as it does not follow such a local property. In this study, we propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues. We revisit the relationships between spatial and Fourier domains and reveal the transform rules on the features of different resolutions in the Fourier domain, which provide key insights for FourierUp's designs. FourierUp as a generic operator consists of three key components: 2D discrete Fourier transform, Fourier dimension increase rules, and 2D inverse Fourier transform, which can be directly integrated with existing networks. Extensive experiments across multiple computer vision tasks, including object detection, image segmentation, image de-raining, image dehazing, and guided image super-resolution, demonstrate the consistent performance gains obtained by introducing our FourierUp.