Trade-offs in Ensembling, Merging and Routing Among Parameter-Efficient Experts

TL;DR

This paper empirically compares ensembling, merging, and routing strategies for combining parameter-efficient language models, revealing that non-uniform methods and routing yield significant performance benefits in multi-task learning.

Contribution

It provides a comprehensive analysis of trade-offs among ensembling, merging, and routing techniques, highlighting the advantages of non-uniform approaches and efficient expert selection methods.

Findings

Non-uniform ensembling and merging improve performance.

Routing offers greater gains than ensembling or merging.

Clustering and greedy selection reduce routing overhead.

Abstract

While large language models (LLMs) fine-tuned with lightweight adapters achieve strong performance across diverse tasks, their performance on individual tasks depends on the fine-tuning strategy. Fusing independently trained models with different strengths has shown promise for multi-task learning through three main strategies: ensembling, which combines outputs from independent models; merging, which fuses model weights via parameter averaging; and routing, which integrates models in an input-dependent fashion. However, many design decisions in these approaches remain understudied, and the relative benefits of more sophisticated ensembling, merging and routing techniques are not fully understood. We empirically evaluate their trade-offs, addressing two key questions: What are the advantages of going beyond uniform ensembling or merging? And does the flexibility of routing justify its complexity? Our findings indicate that non-uniform ensembling and merging improve performance, but routing offers even greater gains. To mitigate the computational cost of routing, we analyze expert selection techniques, showing that clustering and greedy subset selection can maintain reasonable performance with minimal overhead. These insights advance our understanding of model fusion for multi-task learning.