You Augment Me: Exploring ChatGPT-based Data Augmentation for Semantic Code Search

TL;DR

This paper introduces ChatDANCE, a novel data augmentation method using ChatGPT to generate high-quality code search data, significantly improving model performance and enabling the model to self-improve through an augment-filter-retrain strategy.

Contribution

The paper presents a new ChatGPT-based data augmentation approach for semantic code search, including a filtering mechanism and a self-growth strategy for the backbone model, achieving state-of-the-art results.

Findings

Achieves 13.2% improvement in R@1 over baseline.

Effective filtering enhances data quality and model performance.

Model learns more uniform and aligned code-query representations.

Abstract

Code search plays a crucial role in software development, enabling developers to retrieve and reuse code using natural language queries. While the performance of code search models improves with an increase in high-quality data, obtaining such data can be challenging and expensive. Recently, large language models (LLMs) such as ChatGPT have made remarkable progress in both natural and programming language understanding and generation, offering user-friendly interaction via simple prompts. Inspired by these advancements, we propose a novel approach ChatDANCE, which utilizes high-quality and diverse augmented data generated by a large language model and leverages a filtering mechanism to eliminate low-quality augmentations. Specifically, we first propose a set of ChatGPT prompting rules that are specifically designed for source code and queries. Then, we leverage ChatGPT to rewrite code and queries based on the according prompts and then propose a filtering mechanism which trains a cross-encoder from the backbone model UniXcoder to filter out code and query pairs with low matching scores. Finally, we re-train the backbone model using the obtained high-quality augmented data. Experimental results show that ChatDANCE achieves state-of-the-art performance, improving the best baseline by 13.2% (R@1) and 7% (MRR). Surprisingly, we find that this augment-filter-retrain strategy enables the backbone model (UniXcoder) to self-grow. Moreover, extensive experiments show the effectiveness of each component and ChatDANCE has stable performance under different hyperparameter settings. In addition, we conduct qualitative and quantitative analyses to investigate why ChatDANCE works well and find that it learns a more uniform distribution of representations and effectively aligns the code and query spaces.