End-to-End Image Super-Resolution via Deep and Shallow Convolutional Networks

TL;DR

This paper introduces an end-to-end trainable deep CNN for image super-resolution that jointly learns feature extraction, upsampling, and reconstruction, outperforming existing methods by conducting upsampling in feature space and using multi-scale reconstruction.

Contribution

It proposes a novel end-to-end deep CNN that performs upsampling in feature space and employs multi-scale reconstruction, with a new training approach for faster convergence and better results.

Findings

Outperforms state-of-the-art super-resolution methods on standard datasets.

Joint training of deep and shallow networks accelerates convergence and enhances performance.

Ablation studies confirm the effectiveness of multi-scale reconstruction and feature-space upsampling.

Abstract

One impressive advantage of convolutional neural networks (CNNs) is their ability to automatically learn feature representation from raw pixels, eliminating the need for hand-designed procedures. However, recent methods for single image super-resolution (SR) fail to maintain this advantage. They utilize CNNs in two decoupled steps, i.e., first upsampling the low resolution (LR) image to the high resolution (HR) size with hand-designed techniques (e.g., bicubic interpolation), and then applying CNNs on the upsampled LR image to reconstruct HR results. In this paper, we seek an alternative and propose a new image SR method, which jointly learns the feature extraction, upsampling and HR reconstruction modules, yielding a completely end-to-end trainable deep CNN. As opposed to existing approaches, the proposed method conducts upsampling in the latent feature space with filters that are optimized for the task of image SR. In addition, the HR reconstruction is performed in a multi-scale manner to simultaneously incorporate both short- and long-range contextual information, ensuring more accurate restoration of HR images. To facilitate network training, a new training approach is designed, which jointly trains the proposed deep network with a relatively shallow network, leading to faster convergence and more superior performance. The proposed method is extensively evaluated on widely adopted data sets and improves the performance of state-of-the-art methods with a considerable margin. Moreover, in-depth ablation studies are conducted to verify the contribution of different network designs to image SR, providing additional insights for future research.