HyPER-GAN: Hybrid Patch-Based Image-to-Image Translation for Real-Time Photorealism Enhancement

TL;DR

HyPER-GAN is a lightweight, real-time image translation model that enhances photorealism in synthetic images using a hybrid training strategy, outperforming existing methods in speed and quality.

Contribution

The paper introduces HyPER-GAN, a novel hybrid patch-based training approach for real-time photorealism enhancement in image translation tasks.

Findings

Outperforms state-of-the-art lightweight translation methods in latency and realism

Hybrid training improves visual quality and semantic consistency

Achieves real-time inference suitable for training and evaluation

Abstract

Generative models are widely employed to enhance the photorealism of visual synthetic data for training computer vision algorithms. However, they often introduce visual artifacts that degrade the accuracy of these algorithms and require high computational resources, limiting their applicability in real-time training or evaluation scenarios. In this paper, we propose Hybrid Patch Enhanced Realism Generative Adversarial Network (HyPER-GAN), a lightweight image-to-image translation method based on a U-Net-style generator designed for real-time inference. The model is trained using paired synthetic and photorealism-enhanced images, complemented by a hybrid training strategy that incorporates matched patches from real-world images to improve visual realism and semantic consistency. Experimental results demonstrate that HyPER-GAN outperforms state-of-the-art lightweight paired image-to-image translation methods in terms of inference latency, visual realism, and semantic robustness. Moreover, it is illustrated that the proposed hybrid training strategy indeed improves visual quality and semantic consistency compared to training the model solely with paired synthetic and photorealism-enhanced images. Code and pretrained models are publicly available for download at: https://github.com/stefanos50/HyPER-GAN