Ensembling Language Models with Sequential Monte Carlo

TL;DR

This paper introduces a novel sequential Monte Carlo framework for ensembling language models, enabling more effective aggregation of diverse models during decoding and improving structured text generation performance.

Contribution

It proposes a unified SMC-based method for ensembling multiple language models with various aggregation functions, handling mismatched vocabularies and providing better posterior approximations.

Findings

Alternative ensemble strategies outperform probability averaging.

Better posterior approximations lead to improved performance.

The framework supports models with mismatched vocabularies.

Abstract

Practitioners have access to an abundance of language models and prompting strategies for solving many language modeling tasks; yet prior work shows that modeling performance is highly sensitive to both choices. Classical machine learning ensembling techniques offer a principled approach: aggregate predictions from multiple sources to achieve better performance than any single one. However, applying ensembling to language models during decoding is challenging: naively aggregating next-token probabilities yields samples from a locally normalized, biased approximation of the generally intractable ensemble distribution over strings. In this work, we introduce a unified framework for composing $K$ language models into $f$-ensemble distributions for a wide range of functions $f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$. To sample from these distributions, we propose a byte-level sequential Monte Carlo (SMC) algorithm that operates in a shared character space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a family of $f$-ensembles across prompt and model combinations for various structured text generation tasks, highlighting the benefits of alternative aggregation strategies over traditional probability averaging, and showing that better posterior approximations can yield better ensemble performance.