Multi-Curve Translator for High-Resolution Photorealistic Image Translation

TL;DR

The paper introduces the Multi-Curve Translator (MCT), a method that enables high-resolution photorealistic image translation with reduced computational costs by predicting pixel neighborhoods from downsampled images.

Contribution

MCT is a plug-in approach that predicts neighboring pixels for high-resolution images using only downsampled inputs, significantly lowering computational costs while maintaining or improving performance.

Findings

MCT can process 4K images in real-time.

MCT achieves comparable or better results than existing models.

MCT reduces computational costs for high-resolution image translation.

Abstract

The dominant image-to-image translation methods are based on fully convolutional networks, which extract and translate an image's features and then reconstruct the image. However, they have unacceptable computational costs when working with high-resolution images. To this end, we present the Multi-Curve Translator (MCT), which not only predicts the translated pixels for the corresponding input pixels but also for their neighboring pixels. And if a high-resolution image is downsampled to its low-resolution version, the lost pixels are the remaining pixels' neighboring pixels. So MCT makes it possible to feed the network only the downsampled image to perform the mapping for the full-resolution image, which can dramatically lower the computational cost. Besides, MCT is a plug-in approach that utilizes existing base models and requires only replacing their output layers. Experiments demonstrate that the MCT variants can process 4K images in real-time and achieve comparable or even better performance than the base models on various photorealistic image-to-image translation tasks.