Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models

TL;DR

This survey reviews recent collaborative strategies among large language models, categorizing approaches into merging, ensemble, and cooperation, and discusses their applications, challenges, and future research directions in NLP.

Contribution

It provides a comprehensive categorization and analysis of collaborative strategies in LLMs, highlighting their motivations, methods, and potential applications.

Findings

Categorizes collaborative strategies into merging, ensemble, and cooperation.

Highlights the strengths and challenges of each approach.

Outlines future research directions in LLM collaboration.

Abstract

The remarkable success of Large Language Models (LLMs) has ushered natural language processing (NLP) research into a new era. Despite their diverse capabilities, LLMs trained on different corpora exhibit varying strengths and weaknesses, leading to challenges in maximizing their overall efficiency and versatility. To address these challenges, recent studies have explored collaborative strategies for LLMs. This paper provides a comprehensive overview of this emerging research area, highlighting the motivation behind such collaborations. Specifically, we categorize collaborative strategies into three primary approaches: Merging, Ensemble, and Cooperation. Merging involves integrating multiple LLMs in the parameter space. Ensemble combines the outputs of various LLMs. Cooperation} leverages different LLMs to allow full play to their diverse capabilities for specific tasks. We provide in-depth introductions to these methods from different perspectives and discuss their potential applications. Additionally, we outline future research directions, hoping this work will catalyze further studies on LLM collaborations and paving the way for advanced NLP applications.