An Editorial Network for Enhanced Document Summarization

TL;DR

This paper introduces an Editorial Network that combines extractive and abstractive summarization as a post-processing step, mimicking human editing to improve document summaries.

Contribution

It proposes a novel mixed extractive-abstractive approach with a soft-labeling training method, enhancing summarization quality over existing methods.

Findings

Outperforms state-of-the-art extractive and abstractive baselines

Effective soft-labeling training approach demonstrated

Improved summary coherence and relevance

Abstract

We suggest a new idea of Editorial Network - a mixed extractive-abstractive summarization approach, which is applied as a post-processing step over a given sequence of extracted sentences. Our network tries to imitate the decision process of a human editor during summarization. Within such a process, each extracted sentence may be either kept untouched, rephrased or completely rejected. We further suggest an effective way for training the "editor" based on a novel soft-labeling approach. Using the CNN/DailyMail dataset we demonstrate the effectiveness of our approach compared to state-of-the-art extractive-only or abstractive-only baseline methods.

Tables

Table 1. Table 1: Quality evaluation using ROUGE F-measure (ROUGE-1, ROUGE-2, ROUGE-L) on CNN/DailyMail non-annonymized dataset

	R-1	R-2	R-L
Extractive
Lead-3	40.00	17.50	36.20
SummaRuNNer [18]	39.60	16.20	35.30
Refresh [20]	40.00	18.20	36.60
Rnes w/o coherence [28]	41.25	18.87	37.75
BanditSum [6]	41.50	18.70	37.60
Latent [29]	41.05	18.77	37.54
rnn-ext+RL [1]	41.47	18.72	37.76
NeuSum [30]	41.59	19.01	37.98
Abstractive
Pointer-Generator [23]	39.53	17.28	36.38
KIGN+Prediction-guide [15]	38.95	17.12	35.68
Multi-Task(EG+QG) [10]	39.81	17.64	36.54
RL+pg+cbdec [13]	40.66	17.87	37.06
Saliency+Entail. [21]	40.43	18.00	37.10
Inconsistency loss [12]	40.68	17.97	37.13
Bottom-up [9]	41.22	18.68	38.34
rnn-ext+abs+RL [1]	40.04	17.61	37.59
Mixed Extractive-Abstractive
EditNet	41.42	19.03	38.36

Equations

$$d=tanh\left(W_d\bar{e}+b_d\right),$$

$$softmax\left(Vtanh\left(W_c[e_{s_i},a_{s_i},g_{i-1},d]+b_c\right)+b\right),$$

$$g_i=g_{i-1}+tanh\left(W_g{h}_i\right),$$

$$\mathcal{L}(\pi |S)=-\frac{1}{l}\sum _{s_i\in S}\sum _{{\pi }_i\in \{\text{E},\text{A},\text{R}\}}y({\pi }_i)\log p({\pi }_i),$$

$$y({\pi }_i)=\frac{\bar{r}({\pi }_1^{\ast },\dots ,{\pi }_{i-1}^{\ast },{\pi }_i)}{{\sum }_{{\pi }_j\in \{\text{E},\text{A},\text{R}\}}\bar{r}({\pi }_1^{\ast },\dots ,{\pi }_{i-1}^{\ast },{\pi }_j)}$$

Full text

1 Introduction

Automatic text summarizers condense a given piece of text into a shorter version (the summary). This is done while trying to preserve the main essence of the original text and keeping the generated summary as readable as possible.

Existing summarization methods can be classified into two main types, either extractive or abstractive [8]. Extractive methods select and order text fragments (e.g., sentences) from the original text source [2, 5, 6, 7, 18, 29]. Such methods are relatively simpler to develop and keep the extracted fragments untouched, allowing to preserve important parts, e.g., keyphrases, facts, opinions, etc. Yet, extractive summaries tend to be less fluent, coherent and readable and may include superfluous text.

Abstractive methods apply natural language paraphrasing and/or compression on a given text. A common approach is based on the encoder-decoder (seq-to-seq) paradigm [24], with the original text sequence being encoded while the summary is the decoded sequence. While such methods usually generate summaries with better readability, their quality declines over longer textual inputs, which may lead to higher redundancy [22]. Moreover, such methods are sensitive to vocabulary size, making them more difficult to train and generalize [23].

A common approach for handling long text sequences in abstractive settings is through attention mechanisms, which aim to imitate the attentive reading behaviour of humans [3]. Two main types of attention methods may be utilized, either soft or hard. Soft attention methods first locate salient text regions within the input text and then bias the abstraction process to prefer such regions during decoding [4, 9, 12, 19, 15, 21, 25]. On the other hand, hard attention methods perform abstraction only on text regions that were initially selected by some extraction process [1, 18, 17].

Compared to previous works, whose final summary is either entirely extracted or generated using an abstractive process, in this work, we suggest a new idea of “Editorial Network" (EditNet) – a mixed extractive-abstractive summarization approach. A summary generated by EditNet may include sentences that were either extracted, abstracted or of both types. Moreover, per considered sentence, EditNet may decide not to take either of these decisions and completely reject the sentence.

Using the CNN/DailyMail dataset we demonstrate that, EditNet’s summarization quality transcends that of state-of-the-art abstractive-only baselines. EditNet’s summarization quality is also demonstrated to be highly competitive with that of NeuSum [30], which is, to the best of our knowledge, the best performing extractive-only baseline. Yet, while EditNet obtains (more or less) a similar summarization quality to that of NeuSum, compared to the latter which applies only extraction, the former (on average) applies abstraction to the majority of each summary’s extracted sentences.

2 Editorial Network

Figure 1 now depicts the architecture of our proposed Editorial Network-based approach. We apply this approach as a post-processing step over a given summary whose sentences were selected by some extractor. The key idea is to try to imitate the decision process of a human editor who needs to edit the summary so as to enhance its quality.

Let $S$ denote a summary which was extracted from a given text (document) $D$. The editorial process is implemented by iterating over sentences in $S$ according to the selection order of the extractor. For each sentence in $S$, the “editor" may make three possible decisions. The first decision is to keep the extracted sentence untouched (represented by label E in Figure 1). The second alternative is to rephrase the sentence (represented by label A in Figure 1). Such a decision, for example, may represent the editor’s wish to simplify or compress the original source sentence. The last possible decision is to completely reject the sentence (represented by label R in Figure 1). For example, the editor may wish to ignore a superfluous or duplicate information expressed in the current sentence. An example mixed summary generated by our approach is depicted in Figure 2, further emphasizing the various editor’s decisions.

2.1 Implementing the editor’s decisions

For a given sentence $s\in{D}$, we now denote by $s^{e}$ and $s^{a}$ its original (extracted) and paraphrased (abstracted) versions. To obtain $s^{a}$ we use an abstractor, whose details will be shortly explained (see Section 2.2). Let $e_{s}\in\mathbb{R}^{n}$ and $a_{s}\in\mathbb{R}^{n}$ further denote the corresponding sentence representations of $s^{e}$ and $s^{a}$, respectively. Such representations allow to compare both sentence versions on the same grounds.

Recall that, for each sentence $s_{i}\in{S}$ (in order) the editor makes one of the three possible decisions: extract, abstract or reject $s_{i}$. Therefore, the editor may modify summary $S$ by paraphrasing or rejecting some of its sentences, resulting in a mixed extractive-abstractive summary $S^{\prime}$.

Let $l$ be the number of sentences in $S$. In each step $i\in\{1,2,\ldots,l\}$, in order to make an educated decision, the editor considers both sentence representations $e_{s_{i}}$ and $a_{s_{i}}$ as its input, together with two additional auxiliary representations. The first auxiliary representation is that of the whole document $D$ itself, hereinafter denoted $d\in\mathbb{R}^{n}$. Such a representation provides a global context for decision making. Assuming document $D$ has $N$ sentences, let $\bar{e}=\frac{1}{N}\sum\limits_{s\in{D}}^{N}e_{s}$. Following [1, 27], $d$ is then calculated as follows:

[TABLE]

where $W_{d}\in\mathbb{R}^{n\times{n}}$ and $b_{d}\in\mathbb{R}^{n}$ are learnable parameters.

The second auxiliary representation is that of the summary that was generated by the editor so far, denoted at step $i$ as $g_{i-1}\in\mathbb{R}^{n}$, with $g_{0}=\vec{0}$. Such a representation provides a local context for decision making. Given the four representations as an input, the editor’s decision for sentence $s_{i}\in{S}$ is implemented using two fully-connected layers, as follows:

[TABLE]

where $[\cdot]$ denotes the vectors concatenation, $V\in\mathbb{R}^{3\times{m}}$, $W_{c}\in\mathbb{R}^{m\times{4n}}$, $b_{c}\in\mathbb{R}^{m}$ and $b\in\mathbb{R}^{3}$ are learnable parameters.

In each step $i$, therefore, the editor chooses the action $\pi_{i}\in\{\textsf{E},\textsf{A},\textsf{R}\}$ with the highest likelihood (according to Eq. 2), further denoted $p(\pi_{i})$. Upon decision, in case it is either E or A, the editor appends the corresponding sentence version (i.e., either $s_{i}^{e}$ or $s_{i}^{a}$) to $S^{\prime}$; otherwise, the decision is R and sentence $s_{i}$ is discarded. Depending on its decision, the current summary representation is further updated as follows:

[TABLE]

where $W_{g}\in\mathbb{R}^{n\times{n}}$ are learnable parameters, $g_{i-1}$ is the summary representation from the previous decision step; and $h_{i}\in\{e_{s_{i}},a_{s_{i}},\vec{0}\}$, depending on which decision is made.

Such a network architecture allows to capture various complex interactions between the different inputs. For example, the network may learn that given the global context, one of the sentence versions may allow to produce a summary with a better coverage. As another example, based on the interaction between both sentence versions with either of the local or global contexts (and possibly among the last two), the network may learn that both sentence versions may only add superfluous or redundant information to the summary, and therefore, decide to reject both.

2.2 Extractor and Abstractor

As a proof of concept, in this work, we utilize the extractor and abstractor that were previously used in [1], with a slight modification to the latter, motivated by its specific usage within our approach. We now only highlight important aspects of these two sub-components and kindly refer the reader to [1] for the full implementation details.

The extractor of [1] consists of two main sub-components. The first is an encoder which encodes each sentence $s\in{D}$ into $e_{s}$ using an hierarchical representation333Such a representation is basically a combination of a temporal convolutional model followed by a biLSTM encoder.. The second is a sentence selector using a Pointer-Network [26]. For the latter, let $P(s)$ be the selection likelihood of sentence $s$.

The abstractor of [1] is basically a standard encoder-aligner-decoder with a copy mechanism [23]. Yet, instead of applying it directly only on a single given extracted sentence $s_{i}^{e}\in{S}$, we apply it on a “chunk" of three consecutive sentences444The first and last chunks would only have two consecutive sentences. $(s_{-}^{e},s_{i}^{e},s_{+}^{e})$, where $s_{-}^{e}$ and $s_{+}^{e}$ denote the sentence that precedes and succeeds $s_{i}^{e}$ in $D$, respectively. This in turn, allows to generate an abstractive version of $s_{i}^{e}$ (i.e., $s_{i}^{a}$) that benefits from a wider local context. Inspired by previous soft-attention methods, we further utilize the extractor’s sentence selection likelihoods $P(\cdot)$ for enhancing the abstractor’s attention mechanism, as follows. Let $C(w_{j})$ denote the abstractor’s original attention value of a given word $w_{j}$ occurring in $(s_{-}^{e},s_{i}^{e},s_{+}^{e})$; we then recalculate this value to be $C^{\prime}(w_{j})=\frac{C(w_{j})\cdot P(s)}{Z}$, with $w_{j}\in{s}$ and $s\in\{s_{-}^{e},s_{i}^{e},s_{+}^{e}\}$; $Z=\sum_{s^{\prime}\in\{s_{-}^{e},s_{i}^{e},s_{+}^{e}\}}\sum_{w_{j}\in{s^{\prime}}}C(w_{j})\cdot P(s^{\prime})$ denotes the normalization term.

2.3 Sentence representation

Recall that, in order to compare $s_{i}^{e}$ with $s_{i}^{a}$, we need to represent both sentence versions on as similar grounds as possible. To achieve that, we first replace $s_{i}^{e}$ with $s_{i}^{a}$ within the original document $D$. By doing so, we basically treat sentence $s_{i}^{a}$ as if it was an ordinary sentence within $D$, where the rest of the document remains untouched. We then obtain $s_{i}^{a}$’s representation by encoding it using the extractor’s encoder in a similar way in which sentence $s_{i}^{e}$ was originally supposed to be encoded. This results in a representation $a_{s_{i}}$ that provides a comparable alternative to $e_{s_{i}}$, whose encoding is expected to be effected by similar contextual grounds.

2.4 Network training

We conclude this section with the description of how we train the editor using a novel soft labeling approach. Given text $S$ (with $l$ extracted sentences), let $\pi=(\pi_{1},\ldots,\pi_{l})$ denote its editing decisions sequence. We define the following “soft" cross-entropy loss:

[TABLE]

where, for a given sentence $s_{i}\in{S}$, $y(\pi_{i})$ denotes its soft-label for decision.

We next explain how each soft-label $y(\pi_{i})$ is estimated. To this end, we utilize a given summary quality metric $r(S^{\prime})$ which can be used to evaluate the quality of any given summary $S^{\prime}$ (e.g., ROUGE [16]). Overall, for a given text input $S$ with $l$ sentences, there are $3^{l}$ possible summaries $S^{\prime}$ to consider. Let $\pi^{*}=(\pi_{1}^{*},\ldots,\pi_{l}^{*})$ denote the best decision sequence which results in the summary which maximizes $r(\cdot)$. For $i\in\{1,2,\ldots,l\}$, let $\bar{r}(\pi_{1}^{*},\ldots,\pi_{i-1}^{*},\pi_{i})$ denote the average $r(\cdot)$ value obtained by decision sequences that start with the prefix $(\pi_{1}^{*},\ldots,\pi_{i-1}^{*},\pi_{i})$. Based on $\pi^{*}$, the soft label $y(\pi_{i})$ is then calculated555For $i=1$ we have: $\bar{r}(\pi_{1}^{*},\ldots,\pi_{0}^{*},\pi_{1})=\bar{r}(\pi_{1})$. as follows:

[TABLE]

3 Evaluation

3.1 Dataset and Setup

We trained, validated and tested our approach using the non-annonymized version of the CNN/DailyMail dataset [11]. Following [19], we used the story highlights associated with each article as its ground truth summary. We further used the F-measure versions of ROUGE-1 (R-1), ROUGE-2 (R-2) and ROUGE-L (R-L) as our evaluation metrics [16].

The extractor and abstractor were trained similarly to [1] (including the same hyperparameters). The Editorial Network (hereinafter denoted EditNet) was trained according to Section 2.4, using the ADAM optimizer with a learning rate of $10^{-4}$ and a batch size of $32$. To speedup the training time, we precalculated the soft labels (see Eq. 5). Following [6, 27], we set the reward metric to be $r(\cdot)=\alpha R\mbox{-}1(\cdot)+\beta R\mbox{-}2(\cdot)+\gamma R\mbox{-}L(\cdot)$; with $\alpha=0.4$, $\beta=1$ and $\gamma=0.5$, which were further suggested by [27].

We further applied the Teacher-Forcing approach [14] during training, where we considered the true-label instead of the editor’s decision (including when updating $g_{i}$ at each step $i$ according to Eq. 3). Following [1], we set $m=512$ and $n=512$. We trained for $20$ epochs, which has taken about $72$ hours on a single GPU. We chose the best model over the validation set for testing. Finally, all components were implemented in Python $3.6$ using the pytorch $0.4.1$ package.

3.2 Results

Table 1 compares the quality of EditNet with that of several state-of-the-art extractive-only or abstractive-only baselines. This includes the extractor (rnn-ext-RL) and abstractor (rnn-ext-abs-RL) components of [1] that we further utilized for implementing666The rnn-ext-RL extractor results reported in Table 1 are the ones that were reported by [1]. Training the public extractor released by these authors, we obtained the following significantly lower results: R-1:38.43, R-2:18.07 and R-L:35.37. EditNet (see again Section 2.2).

Overall, EditNet provides a highly competitive summary quality, where it outperforms all baselines in the R-2 and R-L metrics. On R-1, EditNet outperforms all abstractive baselines and almost all extractive ones. Interestingly, EditNet’s summarization quality is quite similar to that of NeuSum [30]. Yet, while NeuSum applies an extraction-only approach, summaries generated by EditNet include a mixture of sentences that have been either extracted or abstracted.

On average, $56\%$ and $18\%$ of EditNet’s decisions were to abstract (A) or reject (R), respectively. Moreover, on average, per summary, EditNet keeps only 33% of the original (extracted) sentences, while the rest (67%) are abstracted ones. This demonstrates that, EditNet has a high capability of utilizing abstraction, while being also able to maintain or reject the original extracted text whenever it is estimated to provide the best benefit for the summary’s quality.

4 Conclusion and Future Work

We have shown that instead of solely applying extraction or abstraction, a better choice would be a mixed one. As future work, we plan to evaluate other alternative extractor+abstractor configurations and try to train the network end-to-end. We further plan to explore reinforcement learning (RL) as an alternative decision making approach.