Hi-Reco: High-Fidelity Real-Time Conversational Digital Humans

TL;DR

This paper introduces a high-fidelity, real-time conversational digital human system that combines realistic visuals, expressive speech, and knowledge-grounded dialogue, enabling natural and immersive interactions.

Contribution

It presents an integrated system with novel retrieval-augmented methods and an asynchronous pipeline for seamless, real-time multi-modal digital human interactions.

Findings

Supports emotionally expressive prosody and wake word detection

Achieves minimal latency in multi-modal coordination

Enables context-aware, knowledge-grounded responses

Abstract

High-fidelity digital humans are increasingly used in interactive applications, yet achieving both visual realism and real-time responsiveness remains a major challenge. We present a high-fidelity, real-time conversational digital human system that seamlessly combines a visually realistic 3D avatar, persona-driven expressive speech synthesis, and knowledge-grounded dialogue generation. To support natural and timely interaction, we introduce an asynchronous execution pipeline that coordinates multi-modal components with minimal latency. The system supports advanced features such as wake word detection, emotionally expressive prosody, and highly accurate, context-aware response generation. It leverages novel retrieval-augmented methods, including history augmentation to maintain conversational flow and intent-based routing for efficient knowledge access. Together, these components form an integrated system that enables responsive and believable digital humans, suitable for immersive applications in communication, education, and entertainment.