DiffPhysCam: Differentiable Physics-Based Camera Simulation for Inverse Rendering and Embodied AI

TL;DR

DiffPhysCam is a novel differentiable camera simulator that offers fine-grained control over camera parameters and optical effects, improving synthetic image realism and enabling inverse scene reconstruction for robotics and embodied AI.

Contribution

It introduces a multi-stage pipeline for differentiable camera simulation with enhanced control, optical modeling, and calibration, supporting both forward and inverse rendering tasks.

Findings

Improves robotic perception in synthetic image tasks.

Enables creation of digital twins for scene reconstruction.

Supports navigation of autonomous vehicles using synthetic images.

Abstract

We introduce DiffPhysCam, a differentiable camera simulator designed to support robotics and embodied AI applications by enabling gradient-based optimization in visual perception pipelines. Generating synthetic images that closely mimic those from real cameras is essential for training visual models and enabling end-to-end visuomotor learning. Moreover, differentiable rendering allows inverse reconstruction of real-world scenes as digital twins, facilitating simulation-based robotics training. However, existing virtual cameras offer limited control over intrinsic settings, poorly capture optical artifacts, and lack tunable calibration parameters -- hindering sim-to-real transfer. DiffPhysCam addresses these limitations through a multi-stage pipeline that provides fine-grained control over camera settings, models key optical effects such as defocus blur, and supports calibration with real-world data. It enables both forward rendering for image synthesis and inverse rendering for 3D scene reconstruction, including mesh and material texture optimization. We show that DiffPhysCam enhances robotic perception performance in synthetic image tasks. As an illustrative example, we create a digital twin of a real-world scene using inverse rendering, simulate it in a multi-physics environment, and demonstrate navigation of an autonomous ground vehicle using images generated by DiffPhysCam.