Guided Debugging of Auto-Translated Code Using Differential Testing

TL;DR

tHinter is a debugging tool that identifies translation errors in auto-translated code by comparing functionality through differential testing, significantly reducing debugging effort and improving correction success rates.

Contribution

The paper introduces tHinter, a novel differential testing-based debugging approach specifically designed for auto-translated code from LLMs, enhancing error localization and fixing efficiency.

Findings

Reduces 71% of lines developers review during debugging

Increases likelihood of fixing errors in one query by 59%

Developers find tHinter satisfactory and helpful

Abstract

Large Language Models (LLMs) hold great promise in the task of code translation. However, the lack of explainability complicates the identification of the inevitable translation errors. In this paper, we propose tHinter, a debugging tool to locate translation errors in auto-translated code. The core idea of tHinter is that correctly translated, the source and translated code should present the same functionalities, giving the same output for the same input. Hence, lines in the translated code responsible for output differences are possibly translation errors. First, tHinter employs fuzzing to generate diverse test cases that thoroughly explore the translated code. Then, tHinter relies on a heuristic algorithm to pinpoint translation errors from coverage information and differential testing execution results of those test cases. This heuristic algorithm is designed to leverage both the statistics and the expertise of developers. Comprehensive experiments with real code show its effectiveness. It reduces 71% lines developers need to review during debugging and increases the likelihood of the LLM fixing translation errors in a single query by 59%. Developers generally consider it satisfactory and helpful.