Boosting Source Code Learning with Text-Oriented Data Augmentation: An Empirical Study

TL;DR

This paper investigates the effectiveness of text-oriented data augmentation methods, originally designed for natural language, in improving source code learning tasks, demonstrating their benefits even when syntax is slightly broken.

Contribution

It is the first comprehensive empirical study applying natural language data augmentation techniques to source code learning tasks.

Findings

Certain data augmentation methods improve model accuracy and robustness.

Benefits persist even when source code syntax is slightly altered.

The study covers four code problems and four neural network architectures.

Abstract

Recent studies have demonstrated remarkable advancements in source code learning, which applies deep neural networks (DNNs) to tackle various software engineering tasks. Similar to other DNN-based domains, source code learning also requires massive high-quality training data to achieve the success of these applications. Data augmentation, a technique used to produce additional training data, is widely adopted in other domains (e.g. computer vision). However, the existing practice of data augmentation in source code learning is limited to simple syntax-preserved methods, such as code refactoring. In this paper, considering that source code can also be represented as text data, we take an early step to investigate the effectiveness of data augmentation methods originally designed for natural language texts in the context of source code learning. To this end, we focus on code classification tasks and conduct a comprehensive empirical study across four critical code problems and four DNN architectures to assess the effectiveness of 25 data augmentation methods. Our results reveal specific data augmentation methods that yield more accurate and robust models for source code learning. Additionally, we discover that the data augmentation methods remain beneficial even when they slightly break source code syntax.