Learning Inter- and Intraframe Representations for Non-Lambertian Photometric Stereo

TL;DR

This paper introduces a novel CNN-based framework with hardware and algorithmic innovations for accurate 3D surface normal estimation of non-Lambertian objects using photometric stereo, improving accuracy and robustness.

Contribution

It presents a complete hardware and two-stage CNN architecture that effectively extracts inter- and intraframe features, incorporating object masks to enhance normal estimation for complex, non-Lambertian surfaces.

Findings

Outperforms state-of-the-art methods in accuracy and efficiency.

Effectively handles dark materials and shadows.

Provides detailed surface normal predictions for complex geometries.

Abstract

Photometric stereo provides an important method for high-fidelity 3D reconstruction based on multiple intensity images captured under different illumination directions. In this paper, we present a complete framework, including a multilight source illumination and acquisition hardware system and a two-stage convolutional neural network (CNN) architecture, to construct inter- and intraframe representations for accurate normal estimation of non-Lambertian objects. We experimentally investigate numerous network design alternatives for identifying the optimal scheme to deploy inter- and intraframe feature extraction modules for the photometric stereo problem. Moreover, we propose utilizing the easily obtained object mask to eliminate adverse interference from invalid background regions in intraframe spatial convolutions, thus effectively improving the accuracy of normal estimation for surfaces made of dark materials or with cast shadows. Experimental results demonstrate that the proposed masked two-stage photometric stereo CNN model (MT-PS-CNN) performs favourably against state-of-the-art photometric stereo techniques in terms of both accuracy and efficiency. In addition, the proposed method is capable of predicting accurate and rich surface normal details for non-Lambertian objects of complex geometry and performs stably given inputs captured in both sparse and dense lighting distributions.