Good things come in three: Generating SO Post Titles with Pre-Trained Models, Self Improvement and Post Ranking

TL;DR

This paper introduces FILLER, a novel approach for generating high-quality Stack Overflow post titles by fine-tuning pre-trained language models with self-improvement and post ranking, significantly outperforming existing baselines.

Contribution

The study presents a new training paradigm incorporating self-improvement and post ranking to enhance title generation quality for Stack Overflow posts.

Findings

FILLER outperforms baseline models like Code2Que and GPT3.5-turbo.

The model generates more relevant and high-quality titles according to empirical evaluation.

User studies confirm the superiority of FILLER in relevance and quality.

Abstract

Stack Overflow is a prominent Q and A forum, supporting developers in seeking suitable resources on programming-related matters. Having high-quality question titles is an effective means to attract developers' attention. Unfortunately, this is often underestimated, leaving room for improvement. Research has been conducted, predominantly leveraging pre-trained models to generate titles from code snippets and problem descriptions. Yet, getting high-quality titles is still a challenging task, attributed to both the quality of the input data (e.g., containing noise and ambiguity) and inherent constraints in sequence generation models. In this paper, we present FILLER as a solution to generating Stack Overflow post titles using a fine-tuned language model with self-improvement and post ranking. Our study focuses on enhancing pre-trained language models for generating titles for Stack Overflow posts, employing a training and subsequent fine-tuning paradigm for these models. To this end, we integrate the model's predictions into the training process, enabling it to learn from its errors, thereby lessening the effects of exposure bias. Moreover, we apply a post-ranking method to produce a variety of sample candidates, subsequently selecting the most suitable one. To evaluate FILLER, we perform experiments using benchmark datasets, and the empirical findings indicate that our model provides high-quality recommendations. Moreover, it significantly outperforms all the baselines, including Code2Que, SOTitle, CCBERT, M3NSCT5, and GPT3.5-turbo. A user study also shows that FILLER provides more relevant titles, with respect to SOTitle and GPT3.5-turbo.