Towards Translating Real-World Code with LLMs: A Study of Translating to Rust

TL;DR

This study evaluates the ability of five state-of-the-art LLMs to translate real-world code to Rust, introducing FLOURINE, a tool that verifies translation correctness via differential fuzzing, and analyzing translation success rates and correction capabilities.

Contribution

The paper presents the first comprehensive study of LLM-based code translation to Rust on real-world projects, along with a novel end-to-end translation and verification framework called FLOURINE.

Findings

The best LLM translated 47% of benchmarks successfully.

FLOURINE effectively verifies translation correctness without pre-existing test cases.

Automated feedback strategies improve translation quality and bug fixing.

Abstract

Large language models (LLMs) show promise in code translation - the task of translating code written in one programming language to another language - due to their ability to write code in most programming languages. However, LLM's effectiveness on translating real-world code remains largely unstudied. In this work, we perform the first substantial study on LLM-based translation to Rust by assessing the ability of five state-of-the-art LLMs, GPT4, Claude 3, Claude 2.1, Gemini Pro, and Mixtral. We conduct our study on code extracted from real-world open source projects. To enable our study, we develop FLOURINE, an end-to-end code translation tool that uses differential fuzzing to check if a Rust translation is I/O equivalent to the original source program, eliminating the need for pre-existing test cases. As part of our investigation, we assess both the LLM's ability to produce an initially successful translation, as well as their capacity to fix a previously generated buggy one. If the original and the translated programs are not I/O equivalent, we apply a set of automated feedback strategies, including feedback to the LLM with counterexamples. Our results show that the most successful LLM can translate 47% of our benchmarks, and also provides insights into next steps for improvements.