Physically-based Lighting Generation for Robotic Manipulation

TL;DR

This paper introduces a novel framework that uses physically-based inverse rendering and diffusion models to generate realistic lighting variations for robotic manipulation demonstrations, enhancing imitation learning and enabling new applications.

Contribution

It presents the first framework combining inverse rendering with diffusion models for lighting generation in robotic tasks, improving realism and utility of generated sequences.

Findings

Achieved 38.75% improvement in imitation learning under new lighting conditions.

Enhanced visual quality of generated sequences.

Enabled downstream applications like background and texture generation.

Abstract

In this paper, we propose the first framework that leverages physically-based inverse rendering for novel lighting generation on existing real-world human demonstrations of robotic manipulation tasks. Specifically, inverse rendering decomposes the first frame in each demonstration into geometric (surface normal, depth) and material (albedo, roughness, metallic) properties, which are then used to render appearance changes under different lighting sources. To improve efficiency and maintain consistency across each generated sequence, we fine-tune Stable Video Diffusion on robot execution videos for temporal lighting propagation. We evaluate our framework by measuring the visual quality of the generated sequences, assessing its effectiveness in improving the imitation learning policy performance (38.75\%) under six unseen real-world lighting conditions, and conduct ablation studies on individual modules of the proposed framework. We further showcase three downstream applications enabled by the proposed framework: background generation, object texture generation and distractor positioning. The code for the framework will be made publicly available.