HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization

TL;DR

HETFORMER is a Transformer-based model with sparse attention mechanisms designed for efficient long-text extractive summarization, capturing semantic structures without complex graph procedures.

Contribution

It introduces a heterogeneous graph modeling approach within a Transformer framework for long-text summarization, improving efficiency and performance.

Findings

Achieves state-of-the-art Rouge F1 scores on summarization tasks.

Uses less memory and fewer parameters compared to existing methods.

Effectively models semantic relationships in long texts.

Abstract

To capture the semantic graph structure from raw text, most existing summarization approaches are built on GNNs with a pre-trained model. However, these methods suffer from cumbersome procedures and inefficient computations for long-text documents. To mitigate these issues, this paper proposes HETFORMER, a Transformer-based pre-trained model with multi-granularity sparse attentions for long-text extractive summarization. Specifically, we model different types of semantic nodes in raw text as a potential heterogeneous graph and directly learn heterogeneous relationships (edges) among nodes by Transformer. Extensive experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1 while using less memory and fewer parameters.