SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization

TL;DR

SparseEval introduces a novel sparse optimization approach using gradient descent to efficiently evaluate large language models, significantly reducing computational costs while maintaining high accuracy across benchmarks.

Contribution

It is the first method to apply gradient-based sparse optimization with iterative anchor refinement for LLM evaluation, improving efficiency and robustness.

Findings

Low estimation error across benchmarks

High Kendall's τ indicating strong correlation with full evaluations

Effective in real-world scenarios with reduced computational costs

Abstract

As large language models (LLMs) continue to scale up, their performance on various downstream tasks has significantly improved. However, evaluating their capabilities has become increasingly expensive, as performing inference on a large number of benchmark samples incurs high computational costs. In this paper, we revisit the model-item performance matrix and show that it exhibits sparsity, that representative items can be selected as anchors, and that the task of efficient benchmarking can be formulated as a sparse optimization problem. Based on these insights, we propose SparseEval, a method that, for the first time, adopts gradient descent to optimize anchor weights and employs an iterative refinement strategy for anchor selection. We utilize the representation capacity of MLP to handle sparse optimization and propose the Anchor Importance Score and Candidate Importance Score to evaluate the value of each item for task-aware refinement. Extensive experiments demonstrate the low estimation error and high Kendall's~$\tau$ of our method across a variety of benchmarks, showcasing its superior robustness and practicality in real-world scenarios. Code is available at {https://github.com/taolinzhang/SparseEval}.