Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberation

TL;DR

This paper presents N-Way Self-Evaluating Deliberation, a dynamic, runtime mixture-of-models architecture that constructs composite models from heterogeneous agents, enabling small models to outperform larger ones efficiently.

Contribution

It introduces a novel runtime optimization protocol and deliberation framework that dynamically assembles expert models, improving efficiency and alignment in heterogeneous agent systems.

Findings

Small models (<20B parameters) match or exceed large models (>100B parameters) performance.

The architecture demonstrates improved hardware efficiency and alignment properties.

Empirical validation on benchmarks shows superior performance and safety features.

Abstract

This paper introduces the N-Way Self-Evaluating Deliberation (NSED) protocol, a Runtime Mixture-of-Models (MoM) architecture that constructs emergent composite models from a plurality of distinct expert agents. Unlike traditional Mixture-of-Experts (MoE) which rely on static gating networks, NSED employs a Dynamic Expertise Broker - a runtime optimization engine that treats model selection as a variation of the Knapsack Problem, binding heterogeneous checkpoints to functional roles based on live telemetry and cost constraints. At the execution layer, we formalize deliberation as a Macro-Scale Recurrent Neural Network (RNN), where the consensus state loops back through a semantic forget gate to enable iterative refinement without proportional VRAM scaling. Key components include an orchestration fabric for trustless N-to-N peer review, a Quadratic Voting activation function for non-linear consensus, and a feedback-driven state update. Empirical validation on challenging benchmarks (AIME 2025, LiveCodeBench) demonstrates that this topology allows ensembles of small (less than 20B) consumer-grade models to match or exceed the performance of state-of-the-art 100B+ parameter models, establishing a new hardware arbitrage efficiency frontier. Furthermore, testing on the DarkBench safety suite reveals intrinsic alignment properties, with peer-mediated correction reducing sycophancy scores below that of any individual agent.