ICo3D: An Interactive Conversational 3D Virtual Human

TL;DR

ICo3D introduces a photorealistic, interactive 3D virtual human avatar capable of real-time conversation, combining multi-view capture, advanced modeling, and large language models for immersive applications.

Contribution

The paper presents a novel method for creating animated, conversational 3D avatars with improved photorealism and seamless face-body merging, enabling real-time interactions.

Findings

Real-time conversational 3D avatar demonstrated in multiple use cases

Enhanced photorealism through SWinGS++ and HeadGaS++ models

Successful merging of face and body models without artifacts

Abstract

This work presents Interactive Conversational 3D Virtual Human (ICo3D), a method for generating an interactive, conversational, and photorealistic 3D human avatar. Based on multi-view captures of a subject, we create an animatable 3D face model and a dynamic 3D body model, both rendered by splatting Gaussian primitives. Once merged together, they represent a lifelike virtual human avatar suitable for real-time user interactions. We equip our avatar with an LLM for conversational ability. During conversation, the audio speech of the avatar is used as a driving signal to animate the face model, enabling precise synchronization. We describe improvements to our dynamic Gaussian models that enhance photorealism: SWinGS++ for body reconstruction and HeadGaS++ for face reconstruction, and provide as well a solution to merge the separate face and body models without artifacts. We also present a demo of the complete system, showcasing several use cases of real-time conversation with the 3D avatar. Our approach offers a fully integrated virtual avatar experience, supporting both oral and written form interactions in immersive environments. ICo3D is applicable to a wide range of fields, including gaming, virtual assistance, and personalized education, among others. Project page: https://ico3d.github.io/