Facial 3D Model Registration Under Occlusions With SensiblePoints-based Reinforced Hypothesis Refinement

TL;DR

This paper introduces a novel method for 3D facial model registration under occlusions by synthesizing additional points, employing reinforcement learning, and verifying hypotheses through visual clues, significantly improving registration accuracy.

Contribution

It proposes SensiblePoints and reinforcement learning-based hypothesis refinement to enhance 3D facial registration under occlusion conditions, addressing landmark inaccuracies and limited reliable points.

Findings

High accuracy in facial registration under occlusion

Effective hypothesis verification using visual clues

Robust performance demonstrated in experiments

Abstract

Registering a 3D facial model to a 2D image under occlusion is difficult. First, not all of the detected facial landmarks are accurate under occlusions. Second, the number of reliable landmarks may not be enough to constrain the problem. We propose a method to synthesize additional points (SensiblePoints) to create pose hypotheses. The visual clues extracted from the fiducial points, non-fiducial points, and facial contour are jointly employed to verify the hypotheses. We define a reward function to measure whether the projected dense 3D model is well-aligned with the confidence maps generated by two fully convolutional networks, and use the function to train recurrent policy networks to move the SensiblePoints. The same reward function is employed in testing to select the best hypothesis from a candidate pool of hypotheses. Experimentation demonstrates that the proposed approach is very promising in solving the facial model registration problem under occlusion.