GeRM: A Generative Rendering Model From Physically Realistic to Photorealistic

TL;DR

GeRM is a novel multimodal generative rendering model that effectively transitions physically-based rendering images into photorealistic images using a learned distribution transfer mechanism and multi-condition guidance.

Contribution

It introduces a new P2P transition framework with a distribution transfer vector, a multi-condition ControlNet, and a residual perceptual transfer mechanism for photorealistic rendering.

Findings

Outperforms state-of-the-art methods in image synthesis and editing.

Synthesizes high-quality controllable images from PBR to PRR.

Demonstrates versatility across diverse rendering applications.

Abstract

While physically-based rendering (PBR) simulates light transport that guarantees physical realism, achieving true photorealistic rendering (PRR) demands prohibitive time and labor, and still struggles to capture the intractable richness of the real world. We propose GeRM, the first multimodal generative rendering model to bridge the gap from PBR to PRR (P2P). We formulate this P2P transition by learning a distribution transfer vector (DTV) field to direct the generative process. To achieve this, we introduce a multi-condition ControlNet that synthesizes PBR images and progressively transitions them into PRR images, guided by G-buffers, text prompts, and cues for enhanced regions. To improve the model's grasp of the image distribution shift driven by text prompts, we propose a residual perceptual transfer mechanism to associate text prompts with corresponding targeted modification regions, which more clearly defines the incremental component updates. To supervise this transfer process, we introduce a multi-agent visual language model framework to construct an expert-guided pairwise transfer dataset, named P2P-50K, where each paired sample corresponds to a specific transfer vector in the DTV field. Extensive experiments demonstrate that GeRM synthesizes high-quality controllable images and outperforms state-of-the-art baselines across diverse applications, including PBR and PRR image synthesis and editing.