A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancement

TL;DR

This paper introduces SMCS, a scalable system for multi-LLM collaboration that dynamically selects and enhances LLM responses, outperforming some closed-source models on multiple benchmarks.

Contribution

The paper presents a novel scalable system with retrieval-based selection and exploration-exploitation modules for effective multi-LLM collaboration.

Findings

SMCS outperforms GPT-4.1 and GPT-o3-mini on benchmarks.

Integrating 15 open-source LLMs improves performance.

SMCS exceeds the average of best results with open-source LLMs.

Abstract

Existing multi-LLM collaboration systems often encounter scalability challenges when integrating new LLMs and tasks, leading to suboptimal performance. To address this, we propose SMCS, a Scalable Multi-LLM Collaboration System designed to effectively coordinate multiple open-source LLMs. The system consists of two core components: a Retrieval-based Prior Selection (RPS) module, which dynamically selects the most suitable LLMs for each input, and an Exploration-Exploitation-Driven Posterior Enhancement (EPE) module, which fosters response diversity and selects high-quality outputs through a hybrid scoring mechanism. Experiments on eight mainstream benchmarks validate the effectiveness of our system: by integrating fifteen open-source LLMs, SMCS outperforms prevailing closed-source LLMs, e.g., GPT-4.1(+5.36%) and GPT-o3-mini(+5.28%) across multiple tasks. Remarkably, it even exceeds the average of best results on different datasets with open-source LLMs (+2.86%), significantly advancing the empirical performance frontier of open-source collaboration. The code is released at https://github.com/magent4aci/SMCS.