VINECS: Video-based Neural Character Skinning

TL;DR

VINECS presents an automated method for creating fully rigged, pose-dependent skinned human avatars from multi-view videos, improving over prior methods by eliminating the need for dense 4D scans and enabling explicit mesh generation.

Contribution

It introduces a novel pipeline that learns pose-dependent skinning weights and appearance fields directly from multi-view videos, advancing automation and accuracy in character rigging.

Findings

Outperforms state-of-the-art methods in skinning quality

Does not require dense 4D scans for training

Generates explicit meshes with vertex correspondence

Abstract

Rigging and skinning clothed human avatars is a challenging task and traditionally requires a lot of manual work and expertise. Recent methods addressing it either generalize across different characters or focus on capturing the dynamics of a single character observed under different pose configurations. However, the former methods typically predict solely static skinning weights, which perform poorly for highly articulated poses, and the latter ones either require dense 3D character scans in different poses or cannot generate an explicit mesh with vertex correspondence over time. To address these challenges, we propose a fully automated approach for creating a fully rigged character with pose-dependent skinning weights, which can be solely learned from multi-view video. Therefore, we first acquire a rigged template, which is then statically skinned. Next, a coordinate-based MLP learns a skinning weights field parameterized over the position in a canonical pose space and the respective pose. Moreover, we introduce our pose- and view-dependent appearance field allowing us to differentiably render and supervise the posed mesh using multi-view imagery. We show that our approach outperforms state-of-the-art while not relying on dense 4D scans.