CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models

TL;DR

This paper introduces CodeEval, a comprehensive benchmarking dataset and framework designed to evaluate large language models' Python coding abilities across multiple proficiency levels and problem types, addressing gaps in existing evaluation methods.

Contribution

The paper presents a novel pedagogical benchmarking approach with a multi-dimensional dataset and an open-source evaluation framework for targeted assessment of code-trained LLMs.

Findings

CodeEval covers 24 aspects of Python programming.

RunCodeEval facilitates detailed, automated evaluation.

Benchmarking reveals specific strengths and weaknesses of LLMs.

Abstract

Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities. While models trained in specialized domains like mathematics or coding have demonstrated remarkable advancements in logical reasoning, there remains a significant gap in evaluating their code generation capabilities. Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses, impeding targeted enhancements in models' reasoning abilities to synthesize code. To bridge this gap, our paper introduces an innovative, pedagogical benchmarking method that mirrors the evaluation processes encountered in academic programming courses. We introduce CodeEval, a multi-dimensional benchmark dataset designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming. The dataset covers three proficiency levels - beginner, intermediate, and advanced - and includes both class-based and function-based problem types with detailed problem specifications and comprehensive test suites. To facilitate widespread adoption, we also developed RunCodeEval, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for CodeEval. RunCodeEval handles test execution, context setup, and metrics generation, enabling researchers to quickly obtain detailed insights into model strengths and weaknesses across complexity levels, problem types, and programming categories. This combination enables targeted evaluation and guides improvements in LLMs' programming proficiencies.