Improving Stack Overflow question title generation with copying enhanced CodeBERT model and bi-modal information

TL;DR

This paper introduces CCBERT, a novel deep learning model that leverages bi-modal information and copying mechanisms to improve automatic question title generation for Stack Overflow, outperforming existing models.

Contribution

The paper presents CCBERT, a new encoder-decoder model using CodeBERT and copy attention to enhance question title generation from full question bodies.

Findings

CCBERT outperforms baseline models on a large dataset.

It maintains high performance on code-only and low-resource datasets.

Human evaluation confirms improved readability and relevance.

Abstract

Context: Stack Overflow is very helpful for software developers who are seeking answers to programming problems. Previous studies have shown that a growing number of questions are of low quality and thus obtain less attention from potential answerers. Gao et al. proposed an LSTM-based model (i.e., BiLSTM-CC) to automatically generate question titles from the code snippets to improve the question quality. However, only using the code snippets in the question body cannot provide sufficient information for title generation, and LSTMs cannot capture the long-range dependencies between tokens. Objective: This paper proposes CCBERT, a deep learning based novel model to enhance the performance of question title generation by making full use of the bi-modal information of the entire question body. Method: CCBERT follows the encoder-decoder paradigm and uses CodeBERT to encode the question body into hidden representations, a stacked Transformer decoder to generate predicted tokens, and an additional copy attention layer to refine the output distribution. Both the encoder and decoder perform the multi-head self-attention operation to better capture the long-range dependencies. This paper builds a dataset containing around 200,000 high-quality questions filtered from the data officially published by Stack Overflow to verify the effectiveness of the CCBERT model. Results: CCBERT outperforms all the baseline models on the dataset. Experiments on both code-only and low-resource datasets show the superiority of CCBERT with less performance degradation. The human evaluation also shows the excellent performance of CCBERT concerning both readability and correlation criteria.