SignSplat: Rendering Sign Language via Gaussian Splatting

TL;DR

This paper introduces SignSplat, a Gaussian splatting framework that accurately renders complex sign language motions from few views by leveraging sequence data, regularization, and adaptive Gaussian control, achieving state-of-the-art results.

Contribution

The paper presents a novel Gaussian splatting method with regularization and adaptive control specifically designed for subtle, complex sign language motions from limited multi-view data.

Findings

Achieves state-of-the-art performance on benchmark datasets.

Significantly outperforms existing methods on complex sign language motions.

Demonstrates high fidelity rendering of subtle human motions from few views.

Abstract

State-of-the-art approaches for conditional human body rendering via Gaussian splatting typically focus on simple body motions captured from many views. This is often in the context of dancing or walking. However, for more complex use cases, such as sign language, we care less about large body motion and more about subtle and complex motions of the hands and face. The problems of building high fidelity models are compounded by the complexity of capturing multi-view data of sign. The solution is to make better use of sequence data, ensuring that we can overcome the limited information from only a few views by exploiting temporal variability. Nevertheless, learning from sequence-level data requires extremely accurate and consistent model fitting to ensure that appearance is consistent across complex motions. We focus on how to achieve this, constraining mesh parameters to build an accurate Gaussian splatting framework from few views capable of modelling subtle human motion. We leverage regularization techniques on the Gaussian parameters to mitigate overfitting and rendering artifacts. Additionally, we propose a new adaptive control method to densify Gaussians and prune splat points on the mesh surface. To demonstrate the accuracy of our approach, we render novel sequences of sign language video, building on neural machine translation approaches to sign stitching. On benchmark datasets, our approach achieves state-of-the-art performance; and on highly articulated and complex sign language motion, we significantly outperform competing approaches.