GRAPE: Generalizable and Robust Multi-view Facial Capture

TL;DR

GRAPE is a multi-view facial capture system that generalizes across different camera arrays and is robust to data noise, enabling easier deployment and data collection for facial capture tasks.

Contribution

The paper introduces a novel initialization module and an update-by-disagreement training strategy to enhance generalization and robustness in multi-view facial capture.

Findings

Effective on FaMoS and FaceScape datasets.

Can be used with different camera arrays without retraining.

Improves robustness to scan noise and registration errors.

Abstract

Deep learning-based multi-view facial capture methods have shown impressive accuracy while being several orders of magnitude faster than a traditional mesh registration pipeline. However, the existing systems (e.g. TEMPEH) are strictly restricted to inference on the data captured by the same camera array used to capture their training data. In this study, we aim to improve the generalization ability so that a trained model can be readily used for inference (i.e. capture new data) on a different camera array. To this end, we propose a more generalizable initialization module to extract the camera array-agnostic 3D feature, including a visual hull-based head localization and a visibility-aware 3D feature aggregation module enabled by the visual hull. In addition, we propose an ``update-by-disagreement'' learning strategy to better handle data noise (e.g. inaccurate registration, scan noise) by discarding potentially inaccurate supervision signals during training. The resultant generalizable and robust topologically consistent multi-view facial capture system (GRAPE) can be readily used to capture data on a different camera array, reducing great effort on data collection and processing. Experiments on the FaMoS and FaceScape datasets demonstrate the effectiveness of the proposed method.