CCoE: A Compact and Efficient LLM Framework with Multi-Expert Collaboration for Resource-Limited Settings

TL;DR

The paper introduces CCoE, a modular multi-expert LLM framework that enhances resource efficiency and domain adaptability, achieving high performance with reduced memory and inference costs in resource-limited settings.

Contribution

CCoE is a novel modular framework that integrates domain-specific experts into a shared LLM backbone, enabling efficient multi-domain support with flexible expert collaboration.

Findings

Achieves state-of-the-art performance across five domains.

Reduces memory usage by 61.3% compared to ensemble methods.

Improves inference efficiency by 0.76x over existing multi-expert approaches.

Abstract

Large Language Models (LLMs) have achieved exceptional performance across diverse domains through training on massive datasets. However, scaling LLMs to support multiple downstream domain applications remains a significant challenge, especially under resource constraints. Existing approaches often struggle to balance performance across multiple domains with resource efficiency, limiting their broader applicability. To address this, we introduce the CCoE architecture, a modular framework that seamlessly integrates domain-specific experts into a unified LLM. By leveraging independently trained expert subnetworks on a shared backbone partition, CCoE achieves state-of-the-art performance while significantly reducing the resource requirements for multi-expert deployments. Furthermore, rule-based gating and expert planning in CCoE enable flexible task allocation, promoting expert collaboration to handle complex reasoning tasks. CCoE not only reduces inference costs but also provides a flexible and scalable solution for integrating domain expertise across diverse applications. Experiments on five domains demonstrate that CCoE achieves comparable performance to current domain-specific LLMs. Moreover, compared to existing multi-domain model ensemble methods, CCoE reduces memory usage by 61.3%, while improving inference efficiency by 0.76x over parameter-efficient multi-expert integration approaches.