MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

TL;DR

This paper introduces MHPP, a new challenging dataset for evaluating language models' code generation abilities beyond existing benchmarks, revealing limitations of current models and improving understanding of their true capabilities.

Contribution

The paper presents MHPP, a curated dataset of hard Python problems that better assesses LLMs' reasoning and coding skills, addressing gaps in existing benchmarks.

Findings

Many high-performing models on HumanEval underperform on MHPP.

MHPP uncovers previously unknown limitations of LLMs.

Evaluation pipeline and leaderboard are publicly available.

Abstract

Recent advancements in large language models (LLMs) have greatly improved code generation, specifically at the function level. For instance, GPT-4o has achieved a 91.0\% pass rate on HumanEval. However, this draws into question the adequacy of existing benchmarks in thoroughly assessing function-level code generation capabilities. Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs' code generation capacities due to limitations in quality, difficulty, and granularity. To resolve this, we introduce the Mostly Hard Python Problems (MHPP) dataset, consisting of 210 unique human-curated problems. By focusing on the combination of natural language and code reasoning, MHPP gauges LLMs' abilities to comprehend specifications and restrictions, engage in multi-step reasoning, and apply coding knowledge effectively. Initial evaluations of 26 LLMs using MHPP showed many high-performing models on HumanEval failed to achieve similar success on MHPP. Moreover, MHPP highlighted various previously undiscovered limitations within various LLMs, leading us to believe that it could pave the way for a better understanding of LLMs' capabilities and limitations. MHPP, evaluation pipeline, and leaderboard can be found in https://github.com/SparksofAGI/MHPP.