EnsemW2S: Enhancing Weak-to-Strong Generalization with Large Language Model Ensembles

TL;DR

This paper introduces EnsemW2S, a novel ensemble method that improves weak language models' ability to generalize to complex tasks and supervise stronger models, especially under distributional shifts, with significant empirical gains.

Contribution

EnsemW2S employs a token-level ensemble strategy to iteratively enhance weak experts, enabling better supervision of strong models on both in-distribution and out-of-distribution data.

Findings

Achieved up to 6% improvement on OOD datasets.

Enhanced weak experts' performance by 4% on ID datasets.

Demonstrated effective supervision of strong models with ensemble weak experts.

Abstract

With Large Language Models (LLMs) rapidly approaching and potentially surpassing human-level performance, it has become imperative to develop approaches capable of effectively supervising and enhancing these powerful models using smaller, human-level models exposed to only human-level data. We address this critical weak-to-strong (W2S) generalization challenge by proposing a novel method aimed at improving weak experts, by training on the same limited human-level data, enabling them to generalize to complex, super-human-level tasks. Our approach, called \textbf{EnsemW2S}, employs a token-level ensemble strategy that iteratively combines multiple weak experts, systematically addressing the shortcomings identified in preceding iterations. By continuously refining these weak models, we significantly enhance their collective ability to supervise stronger student models. We extensively evaluate the generalization performance of both the ensemble of weak experts and the subsequent strong student model across in-distribution (ID) and out-of-distribution (OOD) datasets. For OOD, we specifically introduce question difficulty as an additional dimension for defining distributional shifts. Our empirical results demonstrate notable improvements, achieving 4\%, and 3.2\% improvements on ID datasets and, upto 6\% and 2.28\% on OOD datasets for experts and student models respectively, underscoring the effectiveness of our proposed method in advancing W2S generalization.