HDEE: Heterogeneous Domain Expert Ensemble

TL;DR

This paper investigates how introducing heterogeneity in ensemble models of domain experts—varying model sizes and training steps—improves performance across diverse data domains compared to homogeneous ensembles.

Contribution

It demonstrates that heterogeneous ensembles outperform homogeneous ones in most domains by adapting model size and training effort to domain complexity.

Findings

Heterogeneous ensembles achieve lower perplexity in 20 out of 21 domains.

Varying model size and training steps improves domain-specific performance.

Heterogeneity benefits are consistent across included and excluded domains.

Abstract

Training dense LLMs requires enormous amounts of data and centralized compute, which introduces fundamental bottlenecks and ever-growing costs for large models. Several studies aim to reduce this dependency on centralization by reducing the communication overhead of training dense models. Taking this idea of reducing communication overhead to a natural extreme, by training embarrassingly parallelizable ensembles of small independent experts, has been shown to outperform large dense models trained in traditional centralized settings. However, existing studies do not take into account underlying differences amongst data domains and treat them as monolithic, regardless of their underlying complexity, size, or distribution. In this paper, we explore the effects of introducing heterogeneity to these ensembles of domain expert models. Specifically, by allowing models within the ensemble to vary in size--as well as the number of training steps taken depending on the training data's domain--we study the effect heterogeneity has on these ensembles when evaluated against domains included in, and excluded from, the training set. We use the same compute budget to train heterogeneous ensembles and homogeneous baselines for comparison. We show that the heterogeneous ensembles achieve the lowest perplexity scores in $20$ out of the $21$ data domains used in the evaluation. Our code is available at https://github.com/gensyn-ai/hdee.