SCSNet: An Efficient Paradigm for Learning Simultaneously Image Colorization and Super-Resolution

TL;DR

SCSNet is an end-to-end framework that efficiently combines image colorization and super-resolution into a single process, reducing redundancy and improving performance for practical image restoration tasks.

Contribution

The paper introduces SCSNet, a novel unified model with specialized modules for simultaneous colorization and super-resolution, supporting flexible modes and outperforming existing methods.

Findings

Reduces FID scores by 1.8 and 5.1 in automatic and referential modes.

Over 2 times fewer parameters than state-of-the-art methods.

More than 3 times faster in inference speed.

Abstract

In the practical application of restoring low-resolution gray-scale images, we generally need to run three separate processes of image colorization, super-resolution, and dows-sampling operation for the target device. However, this pipeline is redundant and inefficient for the independent processes, and some inner features could have been shared. Therefore, we present an efficient paradigm to perform {S}imultaneously Image {C}olorization and {S}uper-resolution (SCS) and propose an end-to-end SCSNet to achieve this goal. The proposed method consists of two parts: colorization branch for learning color information that employs the proposed plug-and-play \emph{Pyramid Valve Cross Attention} (PVCAttn) module to aggregate feature maps between source and reference images; and super-resolution branch for integrating color and texture information to predict target images, which uses the designed \emph{Continuous Pixel Mapping} (CPM) module to predict high-resolution images at continuous magnification. Furthermore, our SCSNet supports both automatic and referential modes that is more flexible for practical application. Abundant experiments demonstrate the superiority of our method for generating authentic images over state-of-the-art methods, e.g., averagely decreasing FID by 1.8$\downarrow$ and 5.1 $\downarrow$ compared with current best scores for automatic and referential modes, respectively, while owning fewer parameters (more than $\times$2$\downarrow$) and faster running speed (more than $\times$3$\uparrow$).