Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization

TL;DR

Z-Code++ is a novel pre-trained language model optimized for abstractive summarization, utilizing a two-phase pre-training, disentangled attention, and hierarchical encoding to achieve state-of-the-art results across multiple languages and tasks.

Contribution

The paper introduces Z-Code++, a new pre-trained model with innovative techniques like two-phase pre-training and disentangled attention, significantly improving summarization performance especially in low-resource settings.

Findings

Achieves state-of-the-art on 9 out of 13 summarization tasks

Outperforms much larger models like PaLM-540B and GPT-3-175B in zero-shot and few-shot settings

Effective in low-resource and multilingual summarization scenarios

Abstract

This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state of the art encoder-decoder model using three techniques. First, we use a two-phase pre-training process to improve model's performance on low-resource summarization tasks. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Second, we replace self-attention layers in the encoder with disentangled attention layers, where each word is represented using two vectors that encode its content and position, respectively. Third, we use fusion-in-encoder, a simple yet effective method of encoding long sequences in a hierarchical manner. Z-Code++ creates new state of the art on 9 out of 13 text summarization tasks across 5 languages. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum. In zero-shot and few-shot settings, our model substantially outperforms the competing models.