Less Memory, Faster Speed: Refining Self-Attention Module for Image Reconstruction

TL;DR

This paper introduces a refined self-attention module for image reconstruction that reduces computational complexity from quadratic to linear, enabling faster processing and lower memory usage while maintaining effectiveness.

Contribution

The authors propose a novel self-attention module with reduced complexity, adapting non-local operations and connectivity to improve efficiency without sacrificing performance.

Findings

Reduced time complexity from O(n^2) to O(n)

Achieved comparable image reconstruction quality

Faster processing and lower memory usage

Abstract

Self-attention (SA) mechanisms can capture effectively global dependencies in deep neural networks, and have been applied to natural language processing and image processing successfully. However, SA modules for image reconstruction have high time and space complexity, which restrict their applications to higher-resolution images. In this paper, we refine the SA module in self-attention generative adversarial networks (SAGAN) via adapting a non-local operation, revising the connectivity among the units in SA module and re-implementing its computational pattern, such that its time and space complexity is reduced from $\text{O}(n^2)$ to $\text{O}(n)$, but it is still equivalent to the original SA module. Further, we explore the principles behind the module and discover that our module is a special kind of channel attention mechanisms. Experimental results based on two benchmark datasets of image reconstruction, verify that under the same computational environment, two models can achieve comparable effectiveness for image reconstruction, but the proposed one runs faster and takes up less memory space.