REDO: Execution-Free Runtime Error Detection for COding Agents

TL;DR

REDO introduces a novel static analysis approach combining LLMs and tools to detect runtime errors in code without execution, significantly improving error detection accuracy for coding agents.

Contribution

This work presents REDO, a new method integrating LLMs with static analysis for runtime error detection, and introduces SWEDE, a benchmark for evaluating such errors in complex code repositories.

Findings

REDO achieves 11.0% higher accuracy than state-of-the-art methods.

REDO attains 9.1% higher weighted F1 score.

The approach provides better error detection insights for coding agents.

Abstract

As LLM-based agents exhibit exceptional capabilities in addressing complex problems, there is a growing focus on developing coding agents to tackle increasingly sophisticated tasks. Despite their promising performance, these coding agents often produce programs or modifications that contain runtime errors, which can cause code failures and are difficult for static analysis tools to detect. Enhancing the ability of coding agents to statically identify such errors could significantly improve their overall performance. In this work, we introduce Execution-free Runtime Error Detection for COding Agents (REDO), a method that integrates LLMs with static analysis tools to detect runtime errors for coding agents, without code execution. Additionally, we propose a benchmark task, SWE-Bench-Error-Detection (SWEDE), based on SWE-Bench (lite), to evaluate error detection in repository-level problems with complex external dependencies. Finally, through both quantitative and qualitative analyses across various error detection tasks, we demonstrate that REDO outperforms current state-of-the-art methods by achieving a 11.0% higher accuracy and 9.1% higher weighted F1 score; and provide insights into the advantages of incorporating LLMs for error detection.