StreamME: Simplify 3D Gaussian Avatar within Live Stream

TL;DR

StreamME introduces a rapid, on-the-fly 3D avatar reconstruction method from live videos using 3D Gaussian Splatting, enabling real-time applications while preserving privacy and reducing bandwidth.

Contribution

The paper presents a novel fast training strategy for 3D avatar reconstruction that eliminates MLPs and simplifies point cloud distribution for efficiency.

Findings

Achieves real-time 3D avatar reconstruction from live video.

Reduces training time significantly compared to traditional methods.

Supports downstream applications like animation and relighting.

Abstract

We propose StreamME, a method focuses on fast 3D avatar reconstruction. The StreamME synchronously records and reconstructs a head avatar from live video streams without any pre-cached data, enabling seamless integration of the reconstructed appearance into downstream applications. This exceptionally fast training strategy, which we refer to as on-the-fly training, is central to our approach. Our method is built upon 3D Gaussian Splatting (3DGS), eliminating the reliance on MLPs in deformable 3DGS and relying solely on geometry, which significantly improves the adaptation speed to facial expression. To further ensure high efficiency in on-the-fly training, we introduced a simplification strategy based on primary points, which distributes the point clouds more sparsely across the facial surface, optimizing points number while maintaining rendering quality. Leveraging the on-the-fly training capabilities, our method protects the facial privacy and reduces communication bandwidth in VR system or online conference. Additionally, it can be directly applied to downstream application such as animation, toonify, and relighting. Please refer to our project page for more details: https://songluchuan.github.io/StreamME/.