On Predicting the Post-training Potential of Pre-trained LLMs

TL;DR

This paper introduces RuDE, a framework for predicting a base LLM's post-training performance using response discrimination, enabling efficient model selection without extensive training.

Contribution

The paper presents RuDE, a novel approach that accurately forecasts post-training potential of LLMs through a unified, contrastive evaluation framework based on a new taxonomy.

Findings

RuDE achieves over 90% correlation with actual post-training performance.

Validation via RL shows RuDE can identify high-potential small models.

RuDE enables compute-efficient foundation model development.

Abstract

The performance of Large Language Models (LLMs) on downstream tasks is fundamentally constrained by the capabilities acquired during pre-training. However, traditional benchmarks like MMLU often fail to reflect a base model's plasticity in complex open-ended scenarios, leading to inefficient model selection. We address this by introducing a new task of predicting post-training potential - forecasting a base model's performance before post-training. We propose RuDE (Rubric-based Discriminative Evaluation), a unified framework that bypasses the generation gap of base models by leveraging response discrimination. Guided by our systematic 4C Taxonomy, RuDE constructs controlled contrastive pairs across diverse domains by fine-grained rubric violations. Extensive experiments demonstrate a correlation greater than 90% with post-training performance. Crucially, validation via Reinforcement Learning (RL) confirms that RuDE effectively identifies high-potential smaller models that outperform larger counterparts, offering a compute-efficient mechanism for foundation model development.