RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation

TL;DR

RepoTransBench is a comprehensive, real-world benchmark for multilingual repository-level code translation, highlighting current challenges and providing a framework to evaluate and improve large language models' performance on complex, large-scale code translation tasks.

Contribution

The paper introduces RepoTransBench, a novel real-world benchmark with 1,897 repository samples across 13 language pairs, and RepoTransAgent, a general framework for repository-level code translation.

Findings

Best method achieves only 32.8% success rate.

Translation difficulty varies by language pair direction.

Dynamic-to-static translation is more challenging than static-to-dynamic.

Abstract

Repository-level code translation refers to translating an entire code repository from one programming language to another while preserving the functionality of the source repository. Many benchmarks have been proposed to evaluate the performance of such code translators. However, previous benchmarks mostly provide fine-grained samples, focusing at either code snippet, function, or file-level code translation. Such benchmarks do not accurately reflect real-world demands, where entire repositories often need to be translated, involving longer code length and more complex functionalities. To address this gap, we propose a new benchmark, named RepoTransBench, which is a real-world multilingual repository-level code translation benchmark featuring 1,897 real-world repository samples across 13 language pairs with automatically executable test suites. Besides, we introduce RepoTransAgent, a general agent framework to perform repository-level code translation. We evaluate both our benchmark's challenges and agent's effectiveness using several methods and backbone LLMs, revealing that repository-level translation remains challenging, where the best-performing method achieves only a 32.8% success rate. Furthermore, our analysis reveals that translation difficulty varies significantly by language pair direction, with dynamic-to-static language translation being much more challenging than the reverse direction (achieving below 10% vs. static-to-dynamic at 45-63%). Finally, we conduct a detailed error analysis and highlight current LLMs' deficiencies in repository-level code translation, which could provide a reference for further improvements. We provide the code and data at https://github.com/DeepSoftwareAnalytics/RepoTransBench.