Classical Structured Prediction Losses for Sequence to Sequence Learning

TL;DR

This paper explores classical structured prediction losses applied to neural sequence-to-sequence models, demonstrating their competitive performance and achieving state-of-the-art results on translation and summarization tasks.

Contribution

It introduces the application of classical structured prediction losses to neural sequence models, showing they can outperform or match current state-of-the-art methods.

Findings

Classical losses perform well, slightly outperforming beam search optimization.

Achieved new state-of-the-art results on IWSLT'14 German-English translation.

Sequence-level training reaches 41.5 BLEU on WMT'14 English-French.

Abstract

There has been much recent work on training neural attention models at the sequence-level using either reinforcement learning-style methods or by optimizing the beam. In this paper, we survey a range of classical objective functions that have been widely used to train linear models for structured prediction and apply them to neural sequence to sequence models. Our experiments show that these losses can perform surprisingly well by slightly outperforming beam search optimization in a like for like setup. We also report new state of the art results on both IWSLT'14 German-English translation as well as Gigaword abstractive summarization. On the larger WMT'14 English-French translation task, sequence-level training achieves 41.5 BLEU which is on par with the state of the art.