Dynamically Relative Position Encoding-Based Transformer for Automatic Code Edit

TL;DR

This paper introduces DTrans, a Transformer-based model with dynamic relative position encoding, that improves code change prediction accuracy and line localization in automated code editing tasks.

Contribution

The paper proposes a novel Transformer model with dynamic relative position encoding specifically designed for code editing, enhancing long-term dependency modeling and local structure incorporation.

Findings

DTrans outperforms state-of-the-art methods in patch generation accuracy by 5.45%-46.57%.

DTrans achieves 1.75%-24.21% higher accuracy in locating lines to change.

Experiments on benchmark datasets validate the effectiveness of the proposed approach.

Abstract

Adapting Deep Learning (DL) techniques to automate non-trivial coding activities, such as code documentation and defect detection, has been intensively studied recently. Learning to predict code changes is one of the popular and essential investigations. Prior studies have shown that DL techniques such as Neural Machine Translation (NMT) can benefit meaningful code changes, including bug fixing and code refactoring. However, NMT models may encounter bottleneck when modeling long sequences, thus are limited in accurately predicting code changes. In this work, we design a Transformer-based approach, considering that Transformer has proven effective in capturing long-term dependencies. Specifically, we propose a novel model named DTrans. For better incorporating the local structure of code, i.e., statement-level information in this paper, DTrans is designed with dynamically relative position encoding in the multi-head attention of Transformer. Experiments on benchmark datasets demonstrate that DTrans can more accurately generate patches than the state-of-the-art methods, increasing the performance by at least 5.45\%-46.57\% in terms of the exact match metric on different datasets. Moreover, DTrans can locate the lines to change with 1.75\%-24.21\% higher accuracy than the existing methods.