HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair

TL;DR

This paper introduces HEJ-Robust, a benchmark testing the robustness of LLM-based program repair models against syntactic variations, revealing significant performance drops.

Contribution

The creation of HEJ-Robust benchmark with transformations to evaluate and highlight the robustness issues of current LLM-based repair models.

Findings

Model performance drops over 50% under transformations.

Current models lack robustness to minor syntactic variations.

Benchmark exposes robustness gaps in LLM-based program repair.

Abstract

Recent Large Language Models (LLMs) have shown strong performance on automated program repair across standard benchmarks. However, these benchmarks evaluate models on a single canonical form of buggy code and do not reflect the syntactic variations commonly observed in real-world software, leaving robustness largely unexamined. In this work, we construct HEJ-Robust, a robustness benchmark built from HumanEval-Java-Bug using eight semantics-preserving code transformations, resulting in 1,450 transformed instances. We evaluate five fine-tuned LLMs on this benchmark and show that model performance drops by over 50% under several transformations, indicating that current LLM-based repair models lack robustness to minor syntactic variations.