Attentive Fine-Grained Structured Sparsity for Image Restoration

TL;DR

This paper introduces a layer-wise N:M structured sparsity pruning method for image restoration networks, significantly improving efficiency while maintaining high accuracy in tasks like super-resolution and deblurring.

Contribution

It proposes a novel layer-specific pruning ratio determination approach for N:M structured sparsity, enhancing the efficiency-accuracy trade-off in image restoration models.

Findings

Outperforms previous pruning methods in super-resolution and deblurring tasks.

Effectively balances computational efficiency and restoration accuracy.

Demonstrates significant performance gains through extensive experiments.

Abstract

Image restoration tasks have witnessed great performance improvement in recent years by developing large deep models. Despite the outstanding performance, the heavy computation demanded by the deep models has restricted the application of image restoration. To lift the restriction, it is required to reduce the size of the networks while maintaining accuracy. Recently, N:M structured pruning has appeared as one of the effective and practical pruning approaches for making the model efficient with the accuracy constraint. However, it fails to account for different computational complexities and performance requirements for different layers of an image restoration network. To further optimize the trade-off between the efficiency and the restoration accuracy, we propose a novel pruning method that determines the pruning ratio for N:M structured sparsity at each layer. Extensive experimental results on super-resolution and deblurring tasks demonstrate the efficacy of our method which outperforms previous pruning methods significantly. PyTorch implementation for the proposed methods is available at https://github.com/JungHunOh/SLS_CVPR2022.