Less is more: Not all samples are effective for evaluation

TL;DR

This paper introduces a history-free test set compression method for LLM evaluation that significantly reduces computational costs by removing redundant samples without relying on prior model performance data.

Contribution

It proposes a novel domain-adapted embedding and clustering framework with a dataset X-ray mechanism to dynamically calibrate test set compression.

Findings

Reduces evaluation cost by over 90%

Maintains high fidelity to full benchmark results

Effective on large-scale professional-domain datasets

Abstract

The versatility of Large Language Models (LLMs) in vertical domains has spurred the development of numerous specialized evaluation benchmarks. However, these benchmarks often suffer from significant semantic redundancy and impose high computational costs during evaluation. Existing compression methods, such as tinyBenchmarks depend critically on correctness labels from multiple historical models evaluated on the full test set, making them inapplicable in cold-start scenarios, such as the introduction of a new task, domain, or model with no prior evaluation history. To address this limitation, we propose a history-free test set compression framework that requires no prior model performance data. Our method begins by fine-tuning a base LLM on a small amount of domain-specific data to internalize task-relevant semantics. It then generates high-level semantic embeddings for all original test samples using only their raw textual content. In this domain-adapted embedding space, we perform task-aware clustering and introduce a novel dataset X-ray mechanism that analyzes cluster geometry to dynamically calibrate the compression intensity based on the intrinsic redundancy of the benchmark. Experiments on professional-domain dataset, notably a large-scale 3GPP communications benchmark, demonstrate that our approach effectively identifies and removes redundant samples, reducing evaluation cost by over 90% while preserving high fidelity to the full benchmark.