EmpathyEar: An Open-source Avatar Multimodal Empathetic Chatbot
Hao Fei, Han Zhang, Bin Wang, Lizi Liao, Qian Liu, Erik Cambria

TL;DR
EmpathyEar is an open-source multimodal empathetic chatbot that uses advanced language models and multimodal encoders to generate emotionally resonant responses with digital avatars, supporting text, sound, and vision inputs.
Contribution
It introduces a novel open-source avatar-based multimodal empathetic chatbot leveraging large language models and emotion-aware tuning.
Findings
Supports multimodal inputs including text, sound, and vision
Produces digital avatar responses with talking faces and synchronized speech
Achieves deeper emotional resonance in responses
Abstract
This paper introduces EmpathyEar, a pioneering open-source, avatar-based multimodal empathetic chatbot, to fill the gap in traditional text-only empathetic response generation (ERG) systems. Leveraging the advancements of a large language model, combined with multimodal encoders and generators, EmpathyEar supports user inputs in any combination of text, sound, and vision, and produces multimodal empathetic responses, offering users, not just textual responses but also digital avatars with talking faces and synchronized speeches. A series of emotion-aware instruction-tuning is performed for comprehensive emotional understanding and generation capabilities. In this way, EmpathyEar provides users with responses that achieve a deeper emotional resonance, closely emulating human-like empathy. The system paves the way for the next emotional intelligence, for which we open-source the code for…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Taxonomy
TopicsAI in Service Interactions
