Low-Resource Neural Headline Generation

TL;DR

This paper introduces pretraining methods for neural headline generation models that significantly improve performance on small datasets by leveraging all available text and training all model parameters.

Contribution

It presents novel pretraining techniques that enable training all parameters and utilize all text, leading to substantial performance gains on low-resource datasets.

Findings

Up to 32.4% reduction in perplexity

ROUGE score improvement of 2.84 points

Effective pretraining on small datasets

Abstract

Recent neural headline generation models have shown great results, but are generally trained on very large datasets. We focus our efforts on improving headline quality on smaller datasets by the means of pretraining. We propose new methods that enable pre-training all the parameters of the model and utilize all available text, resulting in improvements by up to 32.4% relative in perplexity and 2.84 points in ROUGE.