Refracting Reality: Generating Images with Realistic Transparent Objects

TL;DR

This paper introduces a method to generate images with realistic transparent objects by applying optical physics principles, specifically Snell's Law, to improve the visual plausibility of refraction effects.

Contribution

The authors propose a novel approach that synchronizes pixel rendering with physical refraction laws, enhancing the realism of transparent objects in generated images.

Findings

Generated images show significantly improved optical plausibility.

The method accurately models refraction and reflection effects.

Visual results outperform previous generative models in transparency rendering.

Abstract

Generative image models can produce convincingly real images, with plausible shapes, textures, layouts and lighting. However, one domain in which they perform notably poorly is in the synthesis of transparent objects, which exhibit refraction, reflection, absorption and scattering. Refraction is a particular challenge, because refracted pixel rays often intersect with surfaces observed in other parts of the image, providing a constraint on the color. It is clear from inspection that generative models have not distilled the laws of optics sufficiently well to accurately render refractive objects. In this work, we consider the problem of generating images with accurate refraction, given a text prompt. We synchronize the pixels within the object's boundary with those outside by warping and merging the pixels using Snell's Law of Refraction, at each step of the generation trajectory. For those surfaces that are not directly observed in the image, but are visible via refraction or reflection, we recover their appearance by synchronizing the image with a second generated image -- a panorama centered at the object -- using the same warping and merging procedure. We demonstrate that our approach generates much more optically-plausible images that respect the physical constraints.