Copy that! Editing Sequences by Copying Spans

TL;DR

This paper introduces a span-copying extension to seq2seq models for editing tasks, enabling copying entire spans at once, which improves performance on language and code editing benchmarks.

Contribution

We propose a span-copying mechanism for seq2seq models, along with a new training objective and inference method, to better handle editing tasks involving span copying.

Findings

Our model outperforms baselines on language editing tasks.

Span copying reduces the number of decisions during inference.

The approach improves editing accuracy for natural language and source code.

Abstract

Neural sequence-to-sequence models are finding increasing use in editing of documents, for example in correcting a text document or repairing source code. In this paper, we argue that common seq2seq models (with a facility to copy single tokens) are not a natural fit for such tasks, as they have to explicitly copy each unchanged token. We present an extension of seq2seq models capable of copying entire spans of the input to the output in one step, greatly reducing the number of decisions required during inference. This extension means that there are now many ways of generating the same output, which we handle by deriving a new objective for training and a variation of beam search for inference that explicitly handles this problem. In our experiments on a range of editing tasks of natural language and source code, we show that our new model consistently outperforms simpler baselines.