Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensembling

TL;DR

This paper introduces a novel ensembling method for LLMs by treating token generation as classification, leveraging token-level probabilities to improve accuracy and reduce errors, outperforming existing methods.

Contribution

It proposes a token-level classification ensembling approach for LLMs, exploiting token probabilities to enhance performance and efficiency over traditional full-text ensembling methods.

Findings

Ensembling at token level improves accuracy on benchmarks.

Key token ensembling reduces latency while maintaining performance.

Method surpasses existing community performance ceilings.

Abstract

Ensembling multiple models has always been an effective approach to push the limits of existing performance and is widely used in classification tasks by simply averaging the classification probability vectors from multiple classifiers to achieve better accuracy. However, in the thriving open-source Large Language Model (LLM) community, ensembling methods are rare and typically limited to ensembling the full-text outputs of LLMs, such as selecting the best output using a ranker, which leads to underutilization of token-level probability information. In this paper, we treat the Generation of each token by LLMs as a Classification (GaC) for ensembling. This approach fully exploits the probability information at each generation step and better prevents LLMs from producing early incorrect tokens that lead to snowballing errors. In experiments, we ensemble state-of-the-art LLMs on several benchmarks, including exams, mathematics and reasoning, and observe that our method breaks the existing community performance ceiling. Furthermore, we observed that most of the tokens in the answer are simple and do not affect the correctness of the final answer. Therefore, we also experimented with ensembling only key tokens, and the results showed better performance with lower latency across benchmarks.