Mixture Content Selection for Diverse Sequence Generation

TL;DR

This paper introduces SELECTOR, a plug-and-play module that enhances diversity in sequence generation by explicitly separating content selection from generation, leading to improved accuracy and efficiency in NLP tasks.

Contribution

It proposes a novel mixture of experts approach with stochastic hard-EM training for explicit content diversification in sequence generation models.

Findings

Achieved state-of-the-art top-1 accuracy on question generation and summarization datasets.

Gained a 6% improvement in top-5 accuracy.

Reduced training time by 3.7 times compared to previous models.

Abstract

Generating diverse sequences is important in many NLP applications such as question generation or summarization that exhibit semantically one-to-many relationships between source and the target sequences. We present a method to explicitly separate diversification from generation using a general plug-and-play module (called SELECTOR) that wraps around and guides an existing encoder-decoder model. The diversification stage uses a mixture of experts to sample different binary masks on the source sequence for diverse content selection. The generation stage uses a standard encoder-decoder model given each selected content from the source sequence. Due to the non-differentiable nature of discrete sampling and the lack of ground truth labels for binary mask, we leverage a proxy for ground truth mask and adopt stochastic hard-EM for training. In question generation (SQuAD) and abstractive summarization (CNN-DM), our method demonstrates significant improvements in accuracy, diversity and training efficiency, including state-of-the-art top-1 accuracy in both datasets, 6% gain in top-5 accuracy, and 3.7 times faster training over a state of the art model. Our code is publicly available at https://github.com/clovaai/FocusSeq2Seq.