Perfection Not Required? Human-AI Partnerships in Code Translation

TL;DR

This study investigates software engineers' acceptance of imperfect AI-generated code translations, emphasizing UI features like confidence highlighting to improve understanding and error detection in AI-assisted code modernization.

Contribution

The paper explores engineers' tolerance for imperfect AI code translation and proposes UI enhancements to facilitate effective collaboration with generative NMT models.

Findings

Engineers accept some imperfections in AI-generated code with proper UI cues.

Confidence highlighting and alternate translations improve error detection.

AI assistance is seen as valuable for application modernization tasks.

Abstract

Generative models have become adept at producing artifacts such as images, videos, and prose at human-like levels of proficiency. New generative techniques, such as unsupervised neural machine translation (NMT), have recently been applied to the task of generating source code, translating it from one programming language to another. The artifacts produced in this way may contain imperfections, such as compilation or logical errors. We examine the extent to which software engineers would tolerate such imperfections and explore ways to aid the detection and correction of those errors. Using a design scenario approach, we interviewed 11 software engineers to understand their reactions to the use of an NMT model in the context of application modernization, focusing on the task of translating source code from one language to another. Our three-stage scenario sparked discussions about the utility and desirability of working with an imperfect AI system, how acceptance of that system's outputs would be established, and future opportunities for generative AI in application modernization. Our study highlights how UI features such as confidence highlighting and alternate translations help software engineers work with and better understand generative NMT models.