LaRender: Training-Free Occlusion Control in Image Generation via Latent Rendering

TL;DR

LaRender introduces a training-free, physics-grounded method for precise occlusion control in image generation by leveraging latent space rendering guided by occlusion relationships, outperforming existing approaches.

Contribution

It presents a novel latent space rendering technique that enables explicit occlusion control without retraining diffusion models, based on volume rendering principles.

Findings

Outperforms existing methods in occlusion accuracy

Enables diverse visual effects by adjusting object opacities

Works with pre-trained diffusion models without fine-tuning

Abstract

We propose a novel training-free image generation algorithm that precisely controls the occlusion relationships between objects in an image. Existing image generation methods typically rely on prompts to influence occlusion, which often lack precision. While layout-to-image methods provide control over object locations, they fail to address occlusion relationships explicitly. Given a pre-trained image diffusion model, our method leverages volume rendering principles to "render" the scene in latent space, guided by occlusion relationships and the estimated transmittance of objects. This approach does not require retraining or fine-tuning the image diffusion model, yet it enables accurate occlusion control due to its physics-grounded foundation. In extensive experiments, our method significantly outperforms existing approaches in terms of occlusion accuracy. Furthermore, we demonstrate that by adjusting the opacities of objects or concepts during rendering, our method can achieve a variety of effects, such as altering the transparency of objects, the density of mass (e.g., forests), the concentration of particles (e.g., rain, fog), the intensity of light, and the strength of lens effects, etc.