A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Models

TL;DR

This paper introduces a self-training approach with noise injection sampling to improve neural language generation models, enabling them to produce more semantically accurate utterances for unseen inputs.

Contribution

It proposes an architecture-agnostic self-training method that enhances data with noise-injected samples, significantly improving semantic fidelity in generated utterances.

Findings

Models trained with augmented data produce more semantically correct utterances.

Simple encoder-decoder models achieve state-of-the-art quality after augmentation.

The method improves both automatic and human evaluation metrics.

Abstract

Deep neural networks (DNN) are quickly becoming the de facto standard modeling method for many natural language generation (NLG) tasks. In order for such models to truly be useful, they must be capable of correctly generating utterances for novel meaning representations (MRs) at test time. In practice, even sophisticated DNNs with various forms of semantic control frequently fail to generate utterances faithful to the input MR. In this paper, we propose an architecture agnostic self-training method to sample novel MR/text utterance pairs to augment the original training data. Remarkably, after training on the augmented data, even simple encoder-decoder models with greedy decoding are capable of generating semantically correct utterances that are as good as state-of-the-art outputs in both automatic and human evaluations of quality.