COCO: Testing Code Generation Systems via Concretized Instructions

TL;DR

This paper introduces COCO, a novel testing technique that enhances the robustness evaluation of code generation systems by using concretized instructions, leading to improved detection of robustness issues and aiding in reducing inconsistencies.

Contribution

COCO is a new method that makes instructions more concrete to better test and improve the robustness of code generation systems, outperforming existing testing techniques.

Findings

COCO outperforms existing techniques by over 466%.

Concretized instructions help reduce robustness inconsistencies by up to 53.91%.

Effective on commercial tools like Copilot and ChatGPT.

Abstract

Code generation systems have been extensively developed in recent years to generate source code based on natural language instructions. However, despite their advancements, these systems still face robustness issues where even slightly different instructions can result in significantly different code semantics. Robustness is critical for code generation systems, as it can have significant impacts on software development, software quality, and trust in the generated code. Although existing testing techniques for general text-to-text software can detect some robustness issues, they are limited in effectiveness due to ignoring the characteristics of code generation systems. In this work, we propose a novel technique COCO to test the robustness of code generation systems. It exploits the usage scenario of code generation systems to make the original programming instruction more concrete by incorporating features known to be contained in the original code. A robust system should maintain code semantics for the concretized instruction, and COCO detects robustness inconsistencies when it does not. We evaluated COCO on eight advanced code generation systems, including commercial tools such as Copilot and ChatGPT, using two widely-used datasets. Our results demonstrate the effectiveness of COCO in testing the robustness of code generation systems, outperforming two techniques adopted from general text-to-text software testing by 466.66% and 104.02%, respectively. Furthermore, concretized instructions generated by COCO can help reduce robustness inconsistencies by 18.35% to 53.91% through fine-tuning.