CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping

TL;DR

CrossNet is an end-to-end deep neural network that uses cross-scale warping for reference-based super-resolution, significantly improving alignment precision and computational efficiency over previous methods.

Contribution

The paper introduces CrossNet, a novel fully-convolutional network with cross-scale warping for end-to-end reference-based super-resolution, addressing misalignment and inefficiency issues in prior approaches.

Findings

Achieves 2-4dB higher PSNR than existing methods.

More than 100 times faster in inference.

Improves spatial alignment accuracy at pixel level.

Abstract

The Reference-based Super-resolution (RefSR) super-resolves a low-resolution (LR) image given an external high-resolution (HR) reference image, where the reference image and LR image share similar viewpoint but with significant resolution gap x8. Existing RefSR methods work in a cascaded way such as patch matching followed by synthesis pipeline with two independently defined objective functions, leading to the inter-patch misalignment, grid effect and inefficient optimization. To resolve these issues, we present CrossNet, an end-to-end and fully-convolutional deep neural network using cross-scale warping. Our network contains image encoders, cross-scale warping layers, and fusion decoder: the encoder serves to extract multi-scale features from both the LR and the reference images; the cross-scale warping layers spatially aligns the reference feature map with the LR feature map; the decoder finally aggregates feature maps from both domains to synthesize the HR output. Using cross-scale warping, our network is able to perform spatial alignment at pixel-level in an end-to-end fashion, which improves the existing schemes both in precision (around 2dB-4dB) and efficiency (more than 100 times faster).