Adaptive Patch Exiting for Scalable Single Image Super-Resolution

TL;DR

This paper introduces Adaptive Patch Exiting (APE), a scalable approach for single image super-resolution that allows patches to exit at different network layers based on their difficulty, improving efficiency without sacrificing quality.

Contribution

The paper proposes a novel APE method that predicts patch-specific capacity to enable dynamic exiting, enhancing scalability and practical speedup in super-resolution networks.

Findings

Achieves better speed-performance trade-offs across various backbones.

Demonstrates significant speedup with minimal quality loss.

Validates effectiveness across multiple datasets and scaling factors.

Abstract

Since the future of computing is heterogeneous, scalability is a crucial problem for single image super-resolution. Recent works try to train one network, which can be deployed on platforms with different capacities. However, they rely on the pixel-wise sparse convolution, which is not hardware-friendly and achieves limited practical speedup. As image can be divided into patches, which have various restoration difficulties, we present a scalable method based on Adaptive Patch Exiting (APE) to achieve more practical speedup. Specifically, we propose to train a regressor to predict the incremental capacity of each layer for the patch. Once the incremental capacity is below the threshold, the patch can exit at the specific layer. Our method can easily adjust the trade-off between performance and efficiency by changing the threshold of incremental capacity. Furthermore, we propose a novel strategy to enable the network training of our method. We conduct extensive experiments across various backbones, datasets and scaling factors to demonstrate the advantages of our method. Code is available at https://github.com/littlepure2333/APE