Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Voting

TL;DR

Project Riley introduces a multimodal, multi-agent AI system that simulates emotional reasoning through structured dialogue among emotional agents, integrating visual and textual models for more expressive and context-aware interactions.

Contribution

It presents a novel multi-agent emotional reasoning architecture with multimodal capabilities, including a prototype that enhances emotional expressiveness and factual accuracy in conversational AI.

Findings

Strong emotional alignment in user interactions

High clarity and utility in structured scenarios

Effective integration of multimodal models for reasoning

Abstract

This paper presents Project Riley, a novel multimodal and multi-model conversational AI architecture oriented towards the simulation of reasoning influenced by emotional states. Drawing inspiration from Pixar's Inside Out, the system comprises five distinct emotional agents - Joy, Sadness, Fear, Anger, and Disgust - that engage in structured multi-round dialogues to generate, criticise, and iteratively refine responses. A final reasoning mechanism synthesises the contributions of these agents into a coherent output that either reflects the dominant emotion or integrates multiple perspectives. The architecture incorporates both textual and visual large language models (LLMs), alongside advanced reasoning and self-refinement processes. A functional prototype was deployed locally in an offline environment, optimised for emotional expressiveness and computational efficiency. From this initial prototype, another one emerged, called Armando, which was developed for use in emergency contexts, delivering emotionally calibrated and factually accurate information through the integration of Retrieval-Augmented Generation (RAG) and cumulative context tracking. The Project Riley prototype was evaluated through user testing, in which participants interacted with the chatbot and completed a structured questionnaire assessing three dimensions: Emotional Appropriateness, Clarity and Utility, and Naturalness and Human-likeness. The results indicate strong performance in structured scenarios, particularly with respect to emotional alignment and communicative clarity.