Adaptive Cross-Layer Attention for Image Restoration

TL;DR

This paper introduces an Adaptive Cross-Layer Attention (ACLA) module that enhances image restoration by dynamically aggregating multi-layer features, improving performance across various tasks while maintaining network efficiency.

Contribution

The paper proposes a novel ACLA module with adaptive key selection and insertion search, enabling flexible multi-layer feature aggregation for improved image restoration.

Findings

ACLA improves performance on super-resolution, denoising, demosaicing, and compression artifacts reduction.

ACLA achieves state-of-the-art results with efficient network design.

Extensive experiments validate the effectiveness of adaptive cross-layer attention.

Abstract

Non-local attention module has been proven to be crucial for image restoration. Conventional non-local attention processes features of each layer separately, so it risks missing correlation between features among different layers. To address this problem, we aim to design attention modules that aggregate information from different layers. Instead of finding correlated key pixels within the same layer, each query pixel is encouraged to attend to key pixels at multiple previous layers of the network. In order to efficiently embed such attention design into neural network backbones, we propose a novel Adaptive Cross-Layer Attention (ACLA) module. Two adaptive designs are proposed for ACLA: (1) adaptively selecting the keys for non-local attention at each layer; (2) automatically searching for the insertion locations for ACLA modules. By these two adaptive designs, ACLA dynamically selects a flexible number of keys to be aggregated for non-local attention at previous layer while maintaining a compact neural network with compelling performance. Extensive experiments on image restoration tasks, including single image super-resolution, image denoising, image demosaicing, and image compression artifacts reduction, validate the effectiveness and efficiency of ACLA. The code of ACLA is available at \url{https://github.com/SDL-ASU/ACLA}.