CoLM: Collaborative Large Models via A Client-Server Paradigm

TL;DR

CoLM introduces a client-server framework for large models that enables collaborative reasoning and sharing of high-quality outputs, improving performance on complex tasks and extending beyond language models.

Contribution

The paper proposes CoLM, a novel client-server paradigm for large models that enhances collaboration and performance, especially on challenging queries, and extends to vision-language models.

Findings

Consistent performance improvements on multiple benchmarks.

Effective collaboration enhances single-model reasoning.

Applicable to both language and vision-language models.

Abstract

Large models have achieved remarkable performance across a range of reasoning and understanding tasks. Prior work often utilizes model ensembles or multi-agent systems to collaboratively generate responses, effectively operating in a server-to-server paradigm. However, such approaches do not align well with practical deployment settings, where a limited number of server-side models are shared by many clients under modern internet architectures. In this paper, we introduce \textbf{CoLM} (\textbf{Co}llaboration in \textbf{L}arge-\textbf{M}odels), a novel framework for collaborative reasoning that redefines cooperation among large models from a client-server perspective. Unlike traditional ensemble methods that rely on simultaneous inference from multiple models to produce a single output, CoLM allows the outputs of multiple models to be aggregated or shared, enabling each client model to independently refine and update its own generation based on these high-quality outputs. This design enables collaborative benefits by fully leveraging both client-side and shared server-side models. We further extend CoLM to vision-language models (VLMs), demonstrating its applicability beyond language tasks. Experimental results across multiple benchmarks show that CoLM consistently improves model performance on previously failed queries, highlighting the effectiveness of collaborative guidance in enhancing single-model capabilities.