Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling

TL;DR

This paper investigates the factors affecting large language model ensembling, emphasizing model compatibility, and introduces UniTE, a top-k union method that improves efficiency and performance in model combination.

Contribution

It identifies key determinants of ensemble effectiveness and proposes UniTE, a novel top-k union approach that simplifies model ensembling by avoiding full vocabulary alignment.

Findings

UniTE outperforms existing ensembling methods across benchmarks.

Model compatibility is crucial for effective LLM ensembling.

Top-k union reduces computational costs significantly.

Abstract

Large language models (LLMs) exhibit varying strengths and weaknesses across different tasks, prompting recent studies to explore the benefits of ensembling models to leverage their complementary advantages. However, existing LLM ensembling methods often overlook model compatibility and struggle with inefficient alignment of probabilities across the entire vocabulary. In this study, we empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. This analysis leads to the development of a simple yet effective model selection strategy that identifies compatible models. Additionally, we introduce the \textsc{Uni}on \textsc{T}op-$k$ \textsc{E}nsembling (\textsc{UniTE}), a novel approach that efficiently combines models by focusing on the union of the top-k tokens from each model, thereby avoiding the need for full vocabulary alignment and reducing computational overhead. Extensive evaluations across multiple benchmarks demonstrate that \textsc{UniTE} significantly enhances performance compared to existing methods, offering a more efficient framework for LLM ensembling.