CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models

TL;DR

CodeApex is a bilingual benchmark dataset designed to evaluate large language models' programming comprehension, code generation, and correction abilities, revealing current capabilities and gaps compared to human performance.

Contribution

The paper introduces CodeApex, a comprehensive bilingual benchmark dataset for assessing LLMs' programming skills across multiple tasks, which is novel in scope and focus.

Findings

GPT-4 achieves up to 69% accuracy in programming tasks.

Current LLMs still lag behind human performance.

CodeApex provides a standardized evaluation framework for LLM coding abilities.

Abstract

With the emergence of Large Language Models (LLMs), there has been a significant improvement in the programming capabilities of models, attracting growing attention from researchers. Evaluating the programming capabilities of LLMs is crucial as it reflects the multifaceted abilities of LLMs, and it has numerous downstream applications. In this paper, we propose CodeApex, a bilingual benchmark dataset focusing on the programming comprehension, code generation, and code correction abilities of LLMs. Programming comprehension task tests LLMs on multiple-choice exam questions covering conceptual understanding, commonsense reasoning, and multi-hop reasoning. The code generation task evaluates LLMs through completing C++ functions based on provided descriptions and prototypes. The code correction task asks LLMs to fix real-world erroneous code segments with different error messages. We evaluate 12 widely used LLMs, including both general-purpose and specialized models. GPT-4 exhibits the best programming capabilities, achieving approximate accuracy of 69%, 54%, and 66% on the three tasks, respectively. Compared to human performance, there is still significant room for improvement in LLM programming. We hope that CodeApex can serve as a reference for evaluating the coding capabilities of LLMs, further promoting their development and growth.