AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

TL;DR

AGIEval introduces a benchmark using human-centric exams to evaluate foundation models' abilities, revealing GPT-4's near-human performance on standardized tests and highlighting areas needing improvement.

Contribution

The paper presents AGIEval, a new benchmark based on human exams, to assess foundation models' real-world capabilities beyond artificial datasets.

Findings

GPT-4 achieves 95% on SAT Math and 92.5% on Chinese college English tests.

GPT-4 surpasses average human performance on several standardized exams.

Models show strengths in understanding and knowledge but struggle with complex reasoning.

Abstract

Evaluating the general abilities of foundation models to tackle human-level tasks is a vital aspect of their development and application in the pursuit of Artificial General Intelligence (AGI). Traditional benchmarks, which rely on artificial datasets, may not accurately represent human-level capabilities. In this paper, we introduce AGIEval, a novel benchmark specifically designed to assess foundation model in the context of human-centric standardized exams, such as college entrance exams, law school admission tests, math competitions, and lawyer qualification tests. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003, using this benchmark. Impressively, GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam. This demonstrates the extraordinary performance of contemporary foundation models. In contrast, we also find that GPT-4 is less proficient in tasks that require complex reasoning or specific domain knowledge. Our comprehensive analyses of model capabilities (understanding, knowledge, reasoning, and calculation) reveal these models' strengths and limitations, providing valuable insights into future directions for enhancing their general capabilities. By concentrating on tasks pertinent to human cognition and decision-making, our benchmark delivers a more meaningful and robust evaluation of foundation models' performance in real-world scenarios. The data, code, and all model outputs are released in https://github.com/ruixiangcui/AGIEval.