EchoVoices: Preserving Generational Voices and Memories for Seniors and Children

TL;DR

EchoVoices is a comprehensive system that preserves the voices and memories of seniors and children through advanced speech recognition, synthesis, and memory integration, enabling intergenerational connection and digital legacy creation.

Contribution

The paper introduces a novel end-to-end pipeline combining enhanced speech recognition, adaptive speech synthesis, and memory-driven conversational agents tailored for seniors and children.

Findings

Improved speech recognition accuracy on senior and child datasets

High-fidelity, speaker-aware speech synthesis results

Effective memory system for consistent, personalized interactions

Abstract

Recent breakthroughs in intelligent speech and digital human technologies have primarily targeted mainstream adult users, often overlooking the distinct vocal patterns and interaction styles of seniors and children. These demographics possess distinct vocal characteristics, linguistic styles, and interaction patterns that challenge conventional ASR, TTS, and LLM systems. To address this, we introduce EchoVoices, an end-to-end digital human pipeline dedicated to creating persistent digital personas for seniors and children, ensuring their voices and memories are preserved for future generations. Our system integrates three core innovations: a k-NN-enhanced Whisper model for robust speech recognition of atypical speech; an age-adaptive VITS model for high-fidelity, speaker-aware speech synthesis; and an LLM-driven agent that automatically generates persona cards and leverages a RAG-based memory system for conversational consistency. Our experiments, conducted on the SeniorTalk and ChildMandarin datasets, demonstrate significant improvements in recognition accuracy, synthesis quality, and speaker similarity. EchoVoices provides a comprehensive framework for preserving generational voices, offering a new means of intergenerational connection and the creation of lasting digital legacies.