Revisiting the Role of Natural Language Code Comments in Code Translation

TL;DR

This study empirically demonstrates that natural language code comments, especially those describing overall purpose, significantly improve the accuracy of automated code translation across multiple programming languages, with potential to double performance.

Contribution

It provides the first comprehensive large-scale analysis of comments' impact on code translation, introducing COMMENTRA, a new approach that leverages comments to enhance translation quality.

Findings

Comments describing overall purpose improve translation accuracy

Using comments can potentially double translation performance

Comments have a significant positive impact across multiple languages

Abstract

The advent of large language models (LLMs) has ushered in a new era in automated code translation across programming languages. Since most code-specific LLMs are pretrained on well-commented code from large repositories like GitHub, it is reasonable to hypothesize that natural language code comments could aid in improving translation quality. Despite their potential relevance, comments are largely absent from existing code translation benchmarks, rendering their impact on translation quality inadequately characterised. In this paper, we present a large-scale empirical study evaluating the impact of comments on translation performance. Our analysis involves more than $80,000$ translations, with and without comments, of $1100+$ code samples from two distinct benchmarks covering pairwise translations between five different programming languages: C, C++, Go, Java, and Python. Our results provide strong evidence that code comments, particularly those that describe the overall purpose of the code rather than line-by-line functionality, significantly enhance translation accuracy. Based on these findings, we propose COMMENTRA, a code translation approach, and demonstrate that it can potentially double the performance of LLM-based code translation. To the best of our knowledge, our study is the first in terms of its comprehensiveness, scale, and language coverage on how to improve code translation accuracy using code comments.