What is the best model? Application-driven Evaluation for Large Language Models

TL;DR

This paper introduces A-Eval, a comprehensive benchmark for evaluating large language models based on practical application tasks, providing guidance for selecting the most suitable model while considering cost and performance.

Contribution

The paper presents a new application-driven evaluation benchmark with a categorized dataset and an effective assessment method for large language models.

Findings

Model scale correlates with task performance.

Task difficulty impacts model effectiveness.

Guidelines for optimal model selection are proposed.

Abstract

General large language models enhanced with supervised fine-tuning and reinforcement learning from human feedback are increasingly popular in academia and industry as they generalize foundation models to various practical tasks in a prompt manner. To assist users in selecting the best model in practical application scenarios, i.e., choosing the model that meets the application requirements while minimizing cost, we introduce A-Eval, an application-driven LLMs evaluation benchmark for general large language models. First, we categorize evaluation tasks into five main categories and 27 sub-categories from a practical application perspective. Next, we construct a dataset comprising 678 question-and-answer pairs through a process of collecting, annotating, and reviewing. Then, we design an objective and effective evaluation method and evaluate a series of LLMs of different scales on A-Eval. Finally, we reveal interesting laws regarding model scale and task difficulty level and propose a feasible method for selecting the best model. Through A-Eval, we provide clear empirical and engineer guidance for selecting the best model, reducing barriers to selecting and using LLMs and promoting their application and development. Our benchmark is publicly available at https://github.com/UnicomAI/DataSet/tree/main/TestData/GeneralAbility.