MoCo: A One-Stop Shop for Model Collaboration Research

TL;DR

MoCo is a comprehensive Python library that benchmarks and compares 26 model collaboration methods across diverse tasks, demonstrating significant performance improvements over non-collaborative models.

Contribution

This paper introduces MoCo, the first unified toolkit for executing, benchmarking, and analyzing a wide range of model collaboration algorithms at scale.

Findings

Most collaboration strategies outperform non-collaborative models in 61% of settings

The most effective methods outperform by up to 25.8%

Model collaboration benefits include solving problems where single LMs struggle

Abstract

Advancing beyond single monolithic language models (LMs), recent research increasingly recognizes the importance of model collaboration, where multiple LMs collaborate, compose, and complement each other. Existing research on this topic has mostly been disparate and disconnected, from different research communities, and lacks rigorous comparison. To consolidate existing research and establish model collaboration as a school of thought, we present MoCo: a one-stop Python library of executing, benchmarking, and comparing model collaboration algorithms at scale. MoCo features 26 model collaboration methods, spanning diverse levels of cross-model information exchange such as routing, text, logit, and model parameters. MoCo integrates 25 evaluation datasets spanning reasoning, QA, code, safety, and more, while users could flexibly bring their own data. Extensive experiments with MoCo demonstrate that most collaboration strategies outperform models without collaboration in 61.0% of (model, data) settings on average, with the most effective methods outperforming by up to 25.8%. We further analyze the scaling of model collaboration strategies, the training/inference efficiency of diverse methods, highlight that the collaborative system solves problems where single LMs struggle, and discuss future work in model collaboration, all made possible by MoCo. We envision MoCo as a valuable toolkit to facilitate and turbocharge the quest for an open, modular, decentralized, and collaborative AI future.