RustRepoTrans: Repository-level Code Translation Benchmark Targeting Rust
Guangsheng Ou, Mingwei Liu, Yuxuan Chen, Yanlin Wang, Xin Peng, Zibin Zheng

TL;DR
This paper introduces RustRepoTrans, a new benchmark for repository-level code translation into Rust, evaluating LLMs' ability to handle complex, context-aware, incremental translation tasks across multiple source languages.
Contribution
It presents the first repository-level code translation benchmark targeting Rust, emphasizing incremental translation and context handling, and evaluates seven LLMs' performance on this challenging task.
Findings
DeepSeek-R1 achieves 51.5% Pass@1 on RustRepoTrans.
Performance drops by 22.2% when handling repository context.
Existing models struggle with repository-level, incremental translation scenarios.
Abstract
Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically evaluated using benchmarks like CodeTransOcean and RepoTransBench. However, dependency-free benchmarks fail to capture real-world complexities by focusing primarily on simple function-level translations and overlooking repository-level context (e.g., dependencies). Full-repository translation benchmarks significantly exceed the current capabilities of existing models, resulting in performance bottlenecks that fail to provide actionable insights for guiding model development. Furthermore, existing benchmarks do not account for the scenario of incrementally translating new or modified modules from the source to the target language, which demands careful handling of repository-level contexts such as dependencies, cross-module references, and architectural divergence.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDigital Rights Management and Security
