Better Together? An Evaluation of AI-Supported Code Translation

TL;DR

This study evaluates how AI-generated code translations influence software engineers' productivity and accuracy, revealing benefits and challenges of integrating generative models into coding workflows.

Contribution

It provides empirical insights into the impact of AI-supported code translation on developer error rates and workflow, highlighting the importance of interface design for effective use.

Findings

AI assistance reduces code errors in Java-to-Python translation.

Multiple translations improve process more than quality variation.

AI outputs present complex benefits and challenges for developers.

Abstract

Generative machine learning models have recently been applied to source code, for use cases including translating code between programming languages, creating documentation from code, and auto-completing methods. Yet, state-of-the-art models often produce code that is erroneous or incomplete. In a controlled study with 32 software engineers, we examined whether such imperfect outputs are helpful in the context of Java-to-Python code translation. When aided by the outputs of a code translation model, participants produced code with fewer errors than when working alone. We also examined how the quality and quantity of AI translations affected the work process and quality of outcomes, and observed that providing multiple translations had a larger impact on the translation process than varying the quality of provided translations. Our results tell a complex, nuanced story about the benefits of generative code models and the challenges software engineers face when working with their outputs. Our work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.