Agent-Based Modular Learning for Multimodal Emotion Recognition in Human-Agent Systems

TL;DR

This paper introduces a multi-agent modular framework for multimodal emotion recognition in human-agent systems, enhancing flexibility, scalability, and training efficiency by treating modality encoders and classifiers as autonomous agents.

Contribution

It presents a novel multi-agent architecture that allows modular integration, replacement, and efficient training of multimodal emotion recognition components.

Findings

Supports vision, audio, and text modalities in a proof-of-concept implementation.

Reduces computational overhead during training.

Improves flexibility and scalability of emotion recognition systems.

Abstract

Effective human-agent interaction (HAI) relies on accurate and adaptive perception of human emotional states. While multimodal deep learning models - leveraging facial expressions, speech, and textual cues - offer high accuracy in emotion recognition, their training and maintenance are often computationally intensive and inflexible to modality changes. In this work, we propose a novel multi-agent framework for training multimodal emotion recognition systems, where each modality encoder and the fusion classifier operate as autonomous agents coordinated by a central supervisor. This architecture enables modular integration of new modalities (e.g., audio features via emotion2vec), seamless replacement of outdated components, and reduced computational overhead during training. We demonstrate the feasibility of our approach through a proof-of-concept implementation supporting vision, audio, and text modalities, with the classifier serving as a shared decision-making agent. Our framework not only improves training efficiency but also contributes to the design of more flexible, scalable, and maintainable perception modules for embodied and virtual agents in HAI scenarios.