Relative Overfitting and Accept-Reject Framework

TL;DR

This paper introduces a novel ensemble framework for large language models that leverages model segmentation and the concept of relative overfitting to systematically improve performance across NLP tasks and beyond.

Contribution

It proposes a regular, controllable ensemble approach based on fine-grained model segmentation and introduces the concept of relative overfitting to analyze ensemble performance.

Findings

Ensemble rules are generally effective across diverse benchmarks.

The framework provides a theoretical basis for performance improvement.

Validated on language modeling, long-context tasks, and question-answering.

Abstract

The scaling of Large Language Models (LLMs) currently faces significant challenges. Model assembly is widely considered a promising solution to break through these performance bottlenecks. However, current ensembling methods are primarily guided by the statistical expectation that combining multiple models over large samples will lead to performance gains. We propose an ensemble framework that transitions from such stochastic, sample-dependent methods to a regular, controllable approach based on fine-grained model segmentation. This regularity governs how models are segmented to ensure performance improvement, how the magnitude of this improvement varies with model selection, and what factors determine its theoretical maximum. To formalize this pattern, we introduce the concept of'relative overfitting,' which is derived from the performance discrepancies between constituent models and builds a bridge between ensemble outcomes and the inherent attributes of these models. We detail the patterns of this framework within the domain of NLP and briefly describe its extensibility to other fields, such as computer vision (CV) and AI for science. Our approach was validated using both custom-built and pre-trained mainstream models across diverse benchmarks, including language modeling, long-context tasks, and question-answering (QA). The results indicate that the ensemble rules we proposed are generally effective and that we provide a rigorous proof of these rules in certain experimental scenarios. The proposed framework offers a new perspective for understanding ensemble theory and provides a systematic approach to addressing the performance bottlenecks of LLMs.