B-spline Shape from Motion & Shading: An Automatic Free-form Surface Modeling for Face Reconstruction

TL;DR

This paper introduces a novel method for reconstructing continuous B-spline face surfaces from multi-view images by integrating shape from shading and motion, improving face modeling accuracy in varied conditions.

Contribution

It proposes B-spline Shape from Motion and Shading (BsSfMS), a new approach that jointly optimizes B-spline control points using shading and motion derivatives, with an innovative IMLS algorithm for control point optimization.

Findings

Effective in handling different poses, illuminations, and expressions.

Demonstrates robustness on synthetic and real-world datasets.

Produces smooth, continuous face surface models.

Abstract

Recently, many methods have been proposed for face reconstruction from multiple images, most of which involve fundamental principles of Shape from Shading and Structure from motion. However, a majority of the methods just generate discrete surface model of face. In this paper, B-spline Shape from Motion and Shading (BsSfMS) is proposed to reconstruct continuous B-spline surface for multi-view face images, according to an assumption that shading and motion information in the images contain 1st- and 0th-order derivative of B-spline face respectively. Face surface is expressed as a B-spline surface that can be reconstructed by optimizing B-spline control points. Therefore, normals and 3D feature points computed from shading and motion of images respectively are used as the 1st- and 0th- order derivative information, to be jointly applied in optimizing the B-spline face. Additionally, an IMLS (iterative multi-least-square) algorithm is proposed to handle the difficult control point optimization. Furthermore, synthetic samples and LFW dataset are introduced and conducted to verify the proposed approach, and the experimental results demonstrate the effectiveness with different poses, illuminations, expressions etc., even with wild images.