Training Dynamics for Text Summarization Models

TL;DR

This paper investigates the training dynamics of text summarization models, revealing how different knowledge and behaviors are learned at various stages, and proposes training modifications to enhance factuality and abstractiveness.

Contribution

It provides a detailed analysis of how summarization models learn copying and factual behaviors during fine-tuning, and introduces simple training adjustments to optimize specific model qualities.

Findings

Copying behavior is learned early in training across datasets.

Factual errors like hallucinations are learned later and vary across domains.

Training modifications can improve factuality or abstractiveness.

Abstract

Pre-trained language models (e.g. BART) have shown impressive results when fine-tuned on large summarization datasets. However, little is understood about this fine-tuning process, including what knowledge is retained from pre-training time or how content selection and generation strategies are learnt across iterations. In this work, we analyze the training dynamics for generation models, focusing on summarization. Across different datasets (CNN/DM, XSum, MediaSum) and summary properties, such as abstractiveness and hallucination, we study what the model learns at different stages of its fine-tuning process. We find that a propensity to copy the input is learned early in the training process consistently across all datasets studied. On the other hand, factual errors, such as hallucination of unsupported facts, are learnt in the later stages, though this behavior is more varied across domains. Based on these observations, we explore complementary approaches for modifying training: first, disregarding high-loss tokens that are challenging to learn and second, disregarding low-loss tokens that are learnt very quickly in the latter stages of the training process. We show that these simple training modifications allow us to configure our model to achieve different goals, such as improving factuality or improving abstractiveness.