Hierarchical Phrase-based Sequence-to-Sequence Learning

TL;DR

This paper introduces a hierarchical phrase-based neural transducer that combines a discriminative parser with a seq2seq model, improving translation quality by explicitly modeling phrase hierarchies during training and inference.

Contribution

It presents a novel hierarchical phrase-based neural transducer with two inference modes, integrating a bracketing transduction grammar with seq2seq models for improved translation.

Findings

Both inference modes outperform baselines on small machine translation benchmarks.

The model effectively incorporates hierarchical phrase structures into neural translation.

Decoding with the CKY algorithm enables flexible use of translation rules during inference.

Abstract

We describe a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference. Our approach trains two models: a discriminative parser based on a bracketing transduction grammar whose derivation tree hierarchically aligns source and target phrases, and a neural seq2seq model that learns to translate the aligned phrases one-by-one. We use the same seq2seq model to translate at all phrase scales, which results in two inference modes: one mode in which the parser is discarded and only the seq2seq component is used at the sequence-level, and another in which the parser is combined with the seq2seq model. Decoding in the latter mode is done with the cube-pruned CKY algorithm, which is more involved but can make use of new translation rules during inference. We formalize our model as a source-conditioned synchronous grammar and develop an efficient variational inference algorithm for training. When applied on top of both randomly initialized and pretrained seq2seq models, we find that both inference modes performs well compared to baselines on small scale machine translation benchmarks.