Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation

TL;DR

This paper evaluates the effectiveness of Large Language Models in generating security patches for Java vulnerabilities, revealing limited correctness and highlighting the need for rigorous validation due to prevalent semantic misunderstandings.

Contribution

It introduces a comprehensive analysis framework and the Security Repair Score to assess LLM-generated security patches, providing new insights into their strengths and limitations.

Findings

Only 24.8% of patches are fully correct.

51.4% of patches fail security and functionality.

Vulnerability type influences fix success rates.

Abstract

Large Language Models (LLMs) show promise for Automated Program Repair (APR), yet their effectiveness on security vulnerabilities remains poorly characterized. This study analyzes 319 LLM-generated security patchesacross 64 Java vulnerabilities from the Vul4J benchmark. Using tri-axis evaluation (compilation, security via PoV tests, functionality via test suites), the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality. The dominant failure mode is semantic misunderstanding: LLMs produce syntactically valid code but apply incorrect repair strategies. The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251). Vulnerability type strongly predicts difficulty, with fix rates ranging from 0% (input validation) to 45% (infinite loop). These findings demonstrate that LLM security patches require rigorous validation before deployment.