Physically-Based Photometric Bundle Adjustment in Non-Lambertian Environments

TL;DR

This paper introduces a physically-based photometric bundle adjustment method that accounts for non-Lambertian effects, improving camera pose and 3D geometry estimation in real-world environments with complex reflections.

Contribution

It proposes a novel weighting scheme based on material and illumination estimation to handle photometric inconsistencies in non-Lambertian scenes.

Findings

Outperforms existing PBA methods in accuracy

Introduces a new SLAM dataset for non-Lambertian environments

Effectively models material and illumination variations

Abstract

Photometric bundle adjustment (PBA) is widely used in estimating the camera pose and 3D geometry by assuming a Lambertian world. However, the assumption of photometric consistency is often violated since the non-diffuse reflection is common in real-world environments. The photometric inconsistency significantly affects the reliability of existing PBA methods. To solve this problem, we propose a novel physically-based PBA method. Specifically, we introduce the physically-based weights regarding material, illumination, and light path. These weights distinguish the pixel pairs with different levels of photometric inconsistency. We also design corresponding models for material estimation based on sequential images and illumination estimation based on point clouds. In addition, we establish the first SLAM-related dataset of non-Lambertian scenes with complete ground truth of illumination and material. Extensive experiments demonstrated that our PBA method outperforms existing approaches in accuracy.