Learning More from Less: Unlocking Internal Representations for Benchmark Compression

TL;DR

This paper introduces REPCORE, a method that uses aligned hidden states to create small, representative coresets for benchmarking large language models, enabling accurate performance estimation with fewer models.

Contribution

REPCORE is a novel approach that leverages hidden state alignment to improve benchmark coreset construction, especially when limited source data is available.

Findings

REPCORE achieves high estimation accuracy with as few as ten source models.

Experiments show consistent improvements over output-based baselines in ranking and accuracy.

Spectral analysis reveals that aligned representations encode broad tendencies and reasoning patterns.

Abstract

The prohibitive cost of evaluating Large Language Models (LLMs) necessitates efficient alternatives to full-scale benchmarking. Prevalent approaches address this by identifying a small coreset of items to approximate full-benchmark performance. However, existing methods must estimate a reliable item profile from response patterns across many source models, which becomes statistically unstable when the source pool is small. This dependency is particularly limiting for newly released benchmarks with minimal historical evaluation data. We argue that discrete correctness labels are a lossy view of the model's decision process and fail to capture information encoded in hidden states. To address this, we introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets. Using these subsets for performance extrapolation, REPCORE achieves precise estimation accuracy with as few as ten source models. Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy. Spectral analysis further indicates that the aligned representations contain separable components reflecting broad response tendencies and task-specific reasoning patterns.