Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts

TL;DR

This paper introduces a toolkit for efficiently creating and configuring mixture-of-domain-experts models from trained models or adapters, facilitating low-cost domain specialization in large language models.

Contribution

It provides a practical toolkit and guidance for building MOE models, enabling flexible and cost-effective domain adaptation of large language models.

Findings

Toolkit supports creation of MOE from models or adapters

Extensive testing and guidance provided for architecture design

Public repository available for community use

Abstract

We present a toolkit for creating low-cost Mixture-of-Domain-Experts (MOE) from trained models. The toolkit can be used for creating a mixture from models or from adapters. We perform extensive tests and offer guidance on defining the architecture of the resulting MOE using the toolkit. A public repository is available.