Beyond Benchmarking: A New Paradigm for Evaluation and Assessment of Large Language Models

TL;DR

This paper introduces a new paradigm for evaluating large language models that emphasizes ongoing assessment and diagnosis over traditional benchmarking, aiming for more comprehensive and actionable insights.

Contribution

It proposes the Benchmarking-Evaluation-Assessment paradigm, shifting from static benchmarks to continuous, diagnostic evaluation with optimization guidance.

Findings

Deep attribution of LLM issues enables targeted improvements.

Evaluation content is based on specific task-solving rather than fixed benchmarks.

Provides a framework for ongoing assessment and optimization of LLMs.

Abstract

In current benchmarks for evaluating large language models (LLMs), there are issues such as evaluation content restriction, untimely updates, and lack of optimization guidance. In this paper, we propose a new paradigm for the measurement of LLMs: Benchmarking-Evaluation-Assessment. Our paradigm shifts the "location" of LLM evaluation from the "examination room" to the "hospital". Through conducting a "physical examination" on LLMs, it utilizes specific task-solving as the evaluation content, performs deep attribution of existing problems within LLMs, and provides recommendation for optimization.