Empirical Study on Transformer-based Techniques for Software Engineering

TL;DR

This study evaluates transformer-based models like CodeBERT, CodeGPT, and CodeT5 across key software engineering tasks, revealing insights into their suitability, generalization, and resource efficiency, challenging some common assumptions.

Contribution

It provides a comprehensive empirical comparison of transformer models for code tasks, highlighting the decoder-only model's strengths and the limitations of current datasets and benchmarks.

Findings

CodeGPT excels in generation tasks under SOTA metrics.

Models often do not generalize well across datasets.

Resource consumption does not always correlate with performance.

Abstract

Many Transformer-based pre-trained models for code have been developed and applied to code-related tasks. In this paper, we review the existing literature, examine the suitability of model architectures for different tasks, and look at the generalization ability of models on different datasets, and their resource consumption. We examine three very representative pre-trained models for code: CodeBERT, CodeGPT, and CodeT5, and conduct experiments on the top-4 most targeted software engineering tasks that we found in our literature survey: Code Summarization, Bug Fixing, Bug Detection, and Code Search. In our study, we showcase the capability of decoder-only models (CodeGPT) for specific generation tasks under state-of-the-art evaluation metrics and contest the common belief that the encoder-decoder architecture is optimal for general-purpose coding tasks. Additionally, we found that the most frequently used models are not necessarily the most suitable for certain applications and the developers' needs are not adequately addressed by current research. As well, we found that the benchmark and frequent dataset for Bug Fixing and Code Summarization both fail to enable models to generalize onto other datasets for the same task (the frequent dataset refers to the dataset with the highest frequency used in literature other than the benchmark). We use statistical testing to support our conclusions from experiments. Finally, CodeBERT is highly efficient for understanding tasks, whereas CodeT5's efficiency for generation tasks is in doubt, as the highest resource consumption does not guarantee a consistent better performance on different metrics. We also discuss the numerous practical issues in advancing future research on transformer-based models for code-related tasks.