# Sentence-wise Smooth Regularization for Sequence to Sequence Learning

## TL;DR

This paper introduces a sentence-wise regularization technique for sequence-to-sequence models that promotes smooth prediction probabilities, leading to improved performance in translation and summarization tasks.

## Contribution

The paper proposes a novel regularization method that ensures uniform prediction confidence across target sequences, enhancing sequence quality.

## Key findings

- Outperforms conventional MLE loss on multiple tasks
- Achieves higher BLEU scores on translation benchmarks
- Improves sequence generation quality

## Abstract

Maximum-likelihood estimation (MLE) is widely used in sequence to sequence tasks for model training. It uniformly treats the generation/prediction of each target token as multi-class classification, and yields non-smooth prediction probabilities: in a target sequence, some tokens are predicted with small probabilities while other tokens are with large probabilities. According to our empirical study, we find that the non-smoothness of the probabilities results in low quality of generated sequences. In this paper, we propose a sentence-wise regularization method which aims to output smooth prediction probabilities for all the tokens in the target sequence. Our proposed method can automatically adjust the weights and gradients of each token in one sentence to ensure the predictions in a sequence uniformly well. Experiments on three neural machine translation tasks and one text summarization task show that our method outperforms conventional MLE loss on all these tasks and achieves promising BLEU scores on WMT14 English-German and WMT17 Chinese-English translation task.