VoluMe -- Authentic 3D Video Calls from Live Gaussian Splat Prediction

TL;DR

This paper introduces VoluMe, a real-time method for creating authentic 3D reconstructions from 2D webcam videos, enabling realistic and stable 3D video calls without complex hardware.

Contribution

It presents the first real-time 3D Gaussian reconstruction technique from a single webcam feed that maintains authenticity and stability, advancing accessible 3D videoconferencing.

Findings

Achieves state-of-the-art visual quality and stability in 3D reconstructions.

Enables live 3D video calls using only standard 2D cameras.

Demonstrates practical application in real-time 3D meetings.

Abstract

Virtual 3D meetings offer the potential to enhance copresence, increase engagement and thus improve effectiveness of remote meetings compared to standard 2D video calls. However, representing people in 3D meetings remains a challenge; existing solutions achieve high quality by using complex hardware, making use of fixed appearance via enrolment, or by inverting a pre-trained generative model. These approaches lead to constraints that are unwelcome and ill-fitting for videoconferencing applications. We present the first method to predict 3D Gaussian reconstructions in real time from a single 2D webcam feed, where the 3D representation is not only live and realistic, but also authentic to the input video. By conditioning the 3D representation on each video frame independently, our reconstruction faithfully recreates the input video from the captured viewpoint (a property we call authenticity), while generalizing realistically to novel viewpoints. Additionally, we introduce a stability loss to obtain reconstructions that are temporally stable on video sequences. We show that our method delivers state-of-the-art accuracy in visual quality and stability metrics compared to existing methods, and demonstrate our approach in live one-to-one 3D meetings using only a standard 2D camera and display. This demonstrates that our approach can allow anyone to communicate volumetrically, via a method for 3D videoconferencing that is not only highly accessible, but also realistic and authentic.