Under One Sun: Multi-Object Generative Perception of Materials and Illumination
Nobuo Yoshii, Xinran Nicole Han, Ryo Kawahara, Todd Zickler, Ko Nishino

TL;DR
MultiGP is a novel generative inverse rendering method that disentangles reflectance, texture, and illumination from a single image by leveraging scene-wide lighting consistency, enabling detailed material and lighting recovery.
Contribution
It introduces a multi-object generative perception framework with a cascaded architecture, guidance for diffusion convergence, axial attention, and a texture control network for improved inverse rendering.
Findings
Successfully recovers individual textures and reflectance from single images.
Effectively estimates common scene illumination across multiple objects.
Demonstrates superior performance in material and lighting disentanglement.
Abstract
We introduce Multi-Object Generative Perception (MultiGP), a generative inverse rendering method for stochastic sampling of all radiometric constituents -- reflectance, texture, and illumination -- underlying object appearance from a single image. Our key idea to solve this inherently ambiguous radiometric disentanglement is to leverage the fact that while their texture and reflectance may differ, objects in the same scene are all lit by the same illumination. MultiGP exploits this consensus to produce samples of reflectance, texture, and illumination from a single image of known shapes based on four key technical contributions: a cascaded end-to-end architecture that combines image-space and angular-space disentanglement; Coordinated Guidance for diffusion convergence to a single consistent illumination estimate; Axial Attention applied to facilitate ``cross-talk'' between objects of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputer Graphics and Visualization Techniques · Generative Adversarial Networks and Image Synthesis · 3D Shape Modeling and Analysis
