The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Models

TL;DR

This paper introduces a theoretical scaling law for multi-model LLM ensembles, showing they outperform single models and highlighting the importance of model diversity for performance gains.

Contribution

It proposes the Law of Multi-model Collaboration, a new scaling law predicting ensemble performance limits based on total parameter count and model diversity.

Findings

Multi-model ensembles follow a power-law scaling with parameters.

Ensembles outperform single models in performance and lower loss floors.

Heterogeneous model ensembles yield better scaling than homogeneous ones.

Abstract

Recent advances in large language models (LLMs) have been largely driven by scaling laws for individual models, which predict performance improvements as model parameters and data volume increase. However, the capabilities of any single LLM are inherently bounded. One solution originates from intricate interactions among multiple LLMs, rendering their collective performance surpasses that of any constituent model. Despite the rapid proliferation of multi-model integration techniques such as model routing and post-hoc ensembling, a unifying theoretical framework of performance scaling for multi-model collaboration remains absent. In this work, we propose the Law of Multi-model Collaboration, a scaling law that predicts the performance limits of LLM ensembles based on their aggregated parameter budget. To quantify the intrinsic upper bound of multi-model collaboration, we adopt a method-agnostic formulation and assume an idealized integration oracle where the total cross-entropy loss of each sample is determined by the minimum loss of any model in the model pool. Experimental results reveal that multi-model systems follow a power-law scaling with respect to the total parameter count, exhibiting a more significant improvement trend and a lower theoretical loss floor compared to single model scaling. Moreover, ensembles of heterogeneous model families achieve better performance scaling than those formed within a single model family, indicating that model diversity is a primary driver of collaboration gains. These findings suggest that model collaboration represents a critical axis for extending the intelligence frontier of LLMs.