MigrationBench: Repository-Level Code Migration Benchmark from Java 8

TL;DR

MigrationBench is a comprehensive dataset and evaluation framework for assessing large language models on repository-level code migration from Java 8 to newer LTS versions, facilitating research in automated code modernization.

Contribution

The paper introduces MigrationBench, a new benchmark dataset and evaluation framework specifically for Java code migration tasks, including a novel SD-Feedback approach for LLMs.

Findings

SD-Feedback achieves over 62% success rate with Claude-3.5-Sonnet-v2.

Benchmark includes 5,102 repositories for comprehensive evaluation.

Provides a versatile resource for research in code migration.

Abstract

With the rapid advancement of powerful large language models (LLMs) in recent years, a wide range of software engineering tasks can now be addressed using LLMs, significantly enhancing productivity and scalability. Numerous benchmark datasets have been developed to evaluate the coding capabilities of these models, while they primarily focus on code generation and issue-resolution tasks. In contrast, we introduce a new coding benchmark MigrationBench with a distinct focus: code migration. MigrationBench aims to serve as a comprehensive benchmark for migration from Java $8$ to the latest long-term support (LTS) versions (Java $17$, $21$), including a full dataset and its subset selected with $5,102$ and $300$ repositories respectively. Selected is a representative subset curated for complexity and difficulty, offering a versatile resource to support research in the field of code migration. Additionally, we provide a comprehensive evaluation framework to facilitate rigorous and standardized assessment of LLMs on this challenging task. We further propose SD-Feedback and demonstrate that LLMs can effectively tackle repository-level code migration to Java $17$. For the selected subset with Claude-3.5-Sonnet-v2, SD-Feedback achieves $62.33\%$ and $27.33\%$ success rate (pass@1) for minimal and maximal migration respectively. The benchmark dataset and source code are available at: https://huggingface.co/collections/AmazonScience/migrationbench-68125452fc21a4564b92b6c3 and https://github.com/amazon-science/MigrationBench respectively.