E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Model

TL;DR

E3RG is a multimodal LLM-based system that generates emotionally rich, identity-consistent responses by understanding, retrieving, and producing empathetic content, advancing human-computer emotional interactions.

Contribution

The paper introduces E3RG, a novel multimodal LLM framework that decomposes MERG into understanding, memory retrieval, and response generation, achieving state-of-the-art results without additional training.

Findings

Secured Top-1 in the Avatar-based Multimodal Empathy Challenge

Effective zero-shot and few-shot performance validation

Delivers natural, emotionally rich, and identity-consistent responses

Abstract

Multimodal Empathetic Response Generation (MERG) is crucial for building emotionally intelligent human-computer interactions. Although large language models (LLMs) have improved text-based ERG, challenges remain in handling multimodal emotional content and maintaining identity consistency. Thus, we propose E3RG, an Explicit Emotion-driven Empathetic Response Generation System based on multimodal LLMs which decomposes MERG task into three parts: multimodal empathy understanding, empathy memory retrieval, and multimodal response generation. By integrating advanced expressive speech and video generative models, E3RG delivers natural, emotionally rich, and identity-consistent responses without extra training. Experiments validate the superiority of our system on both zero-shot and few-shot settings, securing Top-1 position in the Avatar-based Multimodal Empathy Challenge on ACM MM 25. Our code is available at https://github.com/RH-Lin/E3RG.