Uncovering Systematic Failures of LLMs in Verifying Code Against Natural Language Specifications

TL;DR

This paper reveals that large language models often fail to accurately verify code against natural language specifications, especially with complex prompts, raising concerns about their reliability in code review tasks.

Contribution

It uncovers systematic failures of LLMs in code verification against requirements and proposes improved prompting strategies to mitigate these issues.

Findings

LLMs frequently misjudge correct code as incorrect or defective.

Complex prompting techniques increase misjudgment rates.

Root causes of misjudgments are analyzed and mitigation strategies are proposed.

Abstract

Large language models (LLMs) have become essential tools in software development, widely used for requirements engineering, code generation and review tasks. Software engineers often rely on LLMs to assess whether system code implementation satisfy task requirements, thereby enhancing code robustness and accuracy. However, it remains unclear whether LLMs can reliably determine whether the code complies fully with the given task descriptions, which is usually natural language specifications. In this paper, we uncover a systematic failure of LLMs in evaluating whether code aligns with natural language requirements. Specifically, with widely used benchmarks, we employ unified prompts to judge code correctness. Our results reveal that LLMs frequently misclassify correct code implementations as either ``not satisfying requirements'' or containing potential defects. Surprisingly, more complex prompting, especially when leveraging prompt engineering techniques involving explanations and proposed corrections, leads to higher misjudgment rate, which highlights the critical reliability issues in using LLMs as code review assistants. We further analyze the root causes of these misjudgments, and propose two improved prompting strategies for mitigation. For the first time, our findings reveals unrecognized limitations in LLMs to match code with requirements. We also offer novel insights and practical guidance for effective use of LLMs in automated code review and task-oriented agent scenarios.