Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization

TL;DR

This paper introduces a framework to evaluate and improve the faithfulness-abstractiveness trade-off in abstractive summarization, demonstrating that learned selection can outperform baseline methods in faithfulness and abstraction levels.

Contribution

The paper proposes a novel framework for analyzing faithfulness-abstractiveness trade-offs and develops a selector model that enhances faithfulness while maintaining higher abstraction.

Findings

The framework effectively characterizes the faithfulness-abstractiveness trade-off.

The selector model achieves higher faithfulness scores in human evaluations.

The system surpasses baseline methods in balancing faithfulness and abstractiveness.

Abstract

Despite recent progress in abstractive summarization, systems still suffer from faithfulness errors. While prior work has proposed models that improve faithfulness, it is unclear whether the improvement comes from an increased level of extractiveness of the model outputs as one naive way to improve faithfulness is to make summarization models more extractive. In this work, we present a framework for evaluating the effective faithfulness of summarization systems, by generating a faithfulnessabstractiveness trade-off curve that serves as a control at different operating points on the abstractiveness spectrum. We then show that the Maximum Likelihood Estimation (MLE) baseline as well as a recently proposed method for improving faithfulness, are both worse than the control at the same level of abstractiveness. Finally, we learn a selector to identify the most faithful and abstractive summary for a given document, and show that this system can attain higher faithfulness scores in human evaluations while being more abstractive than the baseline system on two datasets. Moreover, we show that our system is able to achieve a better faithfulness-abstractiveness trade-off than the control at the same level of abstractiveness.