Joint Learning of Frequency and Spatial Domains for Dense Predictions

TL;DR

This paper introduces a joint learning paradigm that combines frequency and spatial domain learning in neural networks, improving efficiency and performance in dense prediction tasks like depth estimation and segmentation.

Contribution

It proposes a novel joint learning framework that leverages frequency domain learning alongside spatial learning, enhancing global and local feature capture and reducing model parameters.

Findings

Achieves competitive performance without pretraining.

Reduces the number of parameters significantly.

Effective in dense prediction tasks like depth estimation and segmentation.

Abstract

Current artificial neural networks mainly conduct the learning process in the spatial domain but neglect the frequency domain learning. However, the learning course performed in the frequency domain can be more efficient than that in the spatial domain. In this paper, we fully explore frequency domain learning and propose a joint learning paradigm of frequency and spatial domains. This paradigm can take full advantage of the preponderances of frequency learning and spatial learning; specifically, frequency and spatial domain learning can effectively capture global and local information, respectively. Exhaustive experiments on two dense prediction tasks, i.e., self-supervised depth estimation and semantic segmentation, demonstrate that the proposed joint learning paradigm can 1) achieve performance competitive to those of state-of-the-art methods in both depth estimation and semantic segmentation tasks, even without pretraining; and 2) significantly reduce the number of parameters compared to other state-of-the-art methods, which provides more chance to develop real-world applications. We hope that the proposed method can encourage more research in cross-domain learning.