Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models

TL;DR

This paper introduces Variational Prefix Tuning (VPT), a parameter-efficient method that enhances pre-trained code models to generate diverse, accurate summaries, providing multiple options for better code understanding.

Contribution

We propose a novel VPT approach integrating CVAE into pre-trained models to produce diverse summaries without retraining, improving code summarization flexibility and effectiveness.

Findings

VPT significantly increases summary diversity and accuracy.

The method is adaptable across various pre-trained models.

Experimental results outperform existing single-output methods.

Abstract

Recent advancements in source code summarization have leveraged transformer-based pre-trained models, including Large Language Models of Code (LLMCs), to automate and improve the generation of code summaries. However, existing methods often focus on generating a single high-quality summary for a given source code, neglecting scenarios where the generated summary might be inadequate and alternative options are needed. In this paper, we introduce Variational Prefix Tuning (VPT), a novel approach that enhances pre-trained models' ability to generate diverse yet accurate sets of summaries, allowing the user to choose the most suitable one for the given source code. Our method integrates a Conditional Variational Autoencoder (CVAE) framework as a modular component into pre-trained models, enabling us to model the distribution of observed target summaries and sample continuous embeddings to be used as prefixes to steer the generation of diverse outputs during decoding. Importantly, we construct our method in a parameter-efficient manner, eliminating the need for expensive model retraining, especially when using LLMCs. Furthermore, we employ a bi-criteria reranking method to select a subset of generated summaries, optimizing both the diversity and the accuracy of the options presented to users. We present extensive experimental evaluations using widely used datasets and current state-of-the-art pre-trained code summarization models to demonstrate the effectiveness of our approach and its adaptability across models.