MoMoE: A Mixture of Expert Agent Model for Financial Sentiment Analysis

TL;DR

MoMoE introduces a hybrid model combining Mixture-of-Experts architecture with multi-agent collaboration, enhancing language understanding and generation performance through specialized expert pathways and iterative refinement.

Contribution

This work develops a novel hybrid MoE-agent framework that integrates expert routing at neural and agent levels, improving efficiency and task specialization.

Findings

Significant performance improvements on language benchmarks

Effective task decomposition via MoE layers in agents

Enhanced collaborative refinement of outputs

Abstract

We present a novel approach called Mixture of Mixture of Expert (MoMoE) that combines the strengths of Mixture-of-Experts (MoE) architectures with collaborative multi-agent frameworks. By modifying the LLaMA 3.1 8B architecture to incorporate MoE layers in each agent of a layered collaborative structure, we create an ensemble of specialized expert agents that iteratively refine their outputs. Each agent leverages an MoE layer in its final attention block, enabling efficient task decomposition while maintaining computational feasibility. This hybrid approach creates specialized pathways through both the model architecture and the agent collaboration layers. Experimental results demonstrate significant improvements across multiple language understanding and generation benchmarks, highlighting the synergistic benefits of combining expert routing at both the neural and agent levels.