Unsupervised Learning of General-Purpose Embeddings for Code Changes

TL;DR

This paper introduces an unsupervised pre-training method for learning general-purpose code change embeddings, which effectively improve downstream tasks like applying code changes and commit message generation.

Contribution

It presents a novel unsupervised pre-training approach that learns code change representations, enabling better performance on code modification tasks without requiring labeled data.

Findings

Outperforms baseline models by 5.9% in applying code changes

Achieves comparable results to supervised models in commit message generation

Demonstrates the effectiveness of unsupervised pre-training for code change tasks

Abstract

Applying machine learning to tasks that operate with code changes requires their numerical representation. In this work, we propose an approach for obtaining such representations during pre-training and evaluate them on two different downstream tasks - applying changes to code and commit message generation. During pre-training, the model learns to apply the given code change in a correct way. This task requires only code changes themselves, which makes it unsupervised. In the task of applying code changes, our model outperforms baseline models by 5.9 percentage points in accuracy. As for the commit message generation, our model demonstrated the same results as supervised models trained for this specific task, which indicates that it can encode code changes well and can be improved in the future by pre-training on a larger dataset of easily gathered code changes.