SuperCoder2.0: Technical Report on Exploring the feasibility of LLMs as Autonomous Programmer

TL;DR

SuperCoder2.0 is an autonomous AI system for software development that combines hierarchical search, code rewriting, and feedback loops to localize and fix bugs effectively, demonstrating competitive performance on the SWE-bench dataset.

Contribution

It introduces a novel hierarchical search and code editing framework for autonomous programming, integrating retrieval-augmented generation and AST-based rewriting to improve bug localization and fixing.

Findings

Achieves 84.33% correct file localization within top 5 candidates

Resolves 34% of test instances successfully

Placed fourth on the SWE-bench leaderboard

Abstract

We present SuperCoder2.0, an advanced autonomous system designed to enhance software development through artificial intelligence. The system combines an AI-native development approach with intelligent agents to enable fully autonomous coding. Key focus areas include a retry mechanism with error output traceback, comprehensive code rewriting and replacement using Abstract Syntax Tree (ast) parsing to minimize linting issues, code embedding technique for retrieval-augmented generation, and a focus on localizing methods for problem-solving rather than identifying specific line numbers. The methodology employs a three-step hierarchical search space reduction approach for code base navigation and bug localization:utilizing Retrieval Augmented Generation (RAG) and a Repository File Level Map to identify candidate files, (2) narrowing down to the most relevant files using a File Level Schematic Map, and (3) extracting 'relevant locations' within these files. Code editing is performed through a two-part module comprising CodeGeneration and CodeEditing, which generates multiple solutions at different temperature values and replaces entire methods or classes to maintain code integrity. A feedback loop executes repository-level test cases to validate and refine solutions. Experiments conducted on the SWE-bench Lite dataset demonstrate SuperCoder2.0's effectiveness, achieving correct file localization in 84.33% of cases within the top 5 candidates and successfully resolving 34% of test instances. This performance places SuperCoder2.0 fourth globally on the SWE-bench leaderboard. The system's ability to handle diverse repositories and problem types highlights its potential as a versatile tool for autonomous software development. Future work will focus on refining the code editing process and exploring advanced embedding models for improved natural language to code mapping.