DFPE: A Diverse Fingerprint Ensemble for Enhancing LLM Performance

TL;DR

The paper introduces DFPE, an ensemble method that combines multiple LLMs using clustering, filtering, and weighting to improve performance and robustness across diverse language tasks.

Contribution

It presents a novel ensemble approach that leverages model diversity and adaptive weighting to enhance LLM performance on complex benchmarks.

Findings

DFPE outperforms single models by 3% overall accuracy.

DFPE achieves 5% higher accuracy at the discipline level.

The method improves robustness and generalization of LLMs.

Abstract

Large Language Models (LLMs) have shown remarkable capabilities across various natural language processing tasks but often struggle to excel uniformly in diverse or complex domains. We propose a novel ensemble method - Diverse Fingerprint Ensemble (DFPE), which leverages the complementary strengths of multiple LLMs to achieve more robust performance. Our approach involves: (1) clustering models based on response "fingerprints" patterns, (2) applying a quantile-based filtering mechanism to remove underperforming models at a per-subject level, and (3) assigning adaptive weights to remaining models based on their subject-wise validation accuracy. In experiments on the Massive Multitask Language Understanding (MMLU) benchmark, DFPE outperforms the best single model by 3% overall accuracy and 5% in discipline-level accuracy. This method increases the robustness and generalization of LLMs and underscores how model selection, diversity preservation, and performance-driven weighting can effectively address challenging, multi-faceted language understanding tasks.