EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints

TL;DR

EDITOR is a novel edit-based transformer model for neural machine translation that effectively incorporates soft lexical constraints, improving translation quality and decoding speed over previous models.

Contribution

Introduces a repositioning operation in an edit-based transformer, enabling flexible, efficient, and constraint-aware sequence generation for neural machine translation.

Findings

Outperforms Levenshtein Transformer in using soft lexical constraints

Speeds up decoding significantly compared to constrained beam search

Achieves comparable or better translation quality on multiple language pairs

Abstract

We introduce an Edit-Based Transformer with Repositioning (EDITOR), which makes sequence generation flexible by seamlessly allowing users to specify preferences in output lexical choice. Building on recent models for non-autoregressive sequence generation (Gu et al., 2019), EDITOR generates new sequences by iteratively editing hypotheses. It relies on a novel reposition operation designed to disentangle lexical choice from word positioning decisions, while enabling efficient oracles for imitation learning and parallel edits at decoding time. Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer (Gu et al., 2019) while speeding up decoding dramatically compared to constrained beam search (Post and Vilar, 2018). EDITOR also achieves comparable or better translation quality with faster decoding speed than the Levenshtein Transformer on standard Romanian-English, English-German, and English-Japanese machine translation tasks.