REGEN: Real-Time Photorealism Enhancement in Games via a Dual-Stage Generative Network Framework

TL;DR

REGEN is a dual-stage generative framework that enhances photorealism in real-time video game rendering, balancing high visual quality with efficient inference to improve player immersion.

Contribution

It introduces a novel dual-stage approach combining unpaired and paired image translation for real-time photorealism enhancement in games.

Findings

Achieves 12x faster frame rate with comparable or better visual quality

Maintains semantic and temporal consistency in generated frames

Validates effectiveness on Unreal Engine with quantitative metrics

Abstract

Photorealism is an important aspect of modern video games since it can shape player experience and impact immersion, narrative engagement, and visual fidelity. To achieve photorealism, beyond traditional rendering pipelines, generative models have been increasingly adopted as an effective approach for bridging the gap between the visual realism of synthetic and real worlds. However, under real-time constraints of video games, existing generative approaches continue to face a tradeoff between visual quality and runtime efficiency. In this work, we present a framework for enhancing the photorealism of rendered game frames using generative networks. We propose REGEN, which first employs a robust unpaired image-to-image translation model to generate semantically consistent photorealistic frames. These generated frames are then used to create a paired dataset, which transforms the problem to a simpler unpaired image-to-image translation. This enables training with a lightweight method, achieving real-time inference without compromising visual quality. We evaluate REGEN on Unreal Engine, showing, by employing the CMMD metric, that it achieves comparable or slightly improved visual quality compared to the robust method, while improving the frame rate by 12x. Additional experiments also validate that REGEN adheres to the semantic preservation of the initial robust image-to-image translation method and maintains temporal consistency. Code, pre-trained models, and demos for this work are available at: https://github.com/stefanos50/REGEN