A Large-Scale Multi-Length Headline Corpus for Analyzing Length-Constrained Headline Generation Model Evaluation

TL;DR

This paper introduces large-scale Japanese news headline corpora with multiple lengths to evaluate length-controlled headline generation, revealing limitations of current evaluation methods for such systems.

Contribution

It provides new multi-length headline datasets and analyzes evaluation challenges for length-constrained headline generation models.

Findings

Longest reference summaries can evaluate existing methods

Current evaluation settings have notable limitations

New corpora enable better assessment of length control in headlines

Abstract

Browsing news articles on multiple devices is now possible. The lengths of news article headlines have precise upper bounds, dictated by the size of the display of the relevant device or interface. Therefore, controlling the length of headlines is essential when applying the task of headline generation to news production. However, because there is no corpus of headlines of multiple lengths for a given article, previous research on controlling output length in headline generation has not discussed whether the system outputs could be adequately evaluated without multiple references of different lengths. In this paper, we introduce two corpora, which are Japanese News Corpus (JNC) and JApanese MUlti-Length Headline Corpus (JAMUL), to confirm the validity of previous evaluation settings. The JNC provides common supervision data for headline generation. The JAMUL is a large-scale evaluation dataset for headlines of three different lengths composed by professional editors. We report new findings on these corpora; for example, although the longest length reference summary can appropriately evaluate the existing methods controlling output length, this evaluation setting has several problems.

Tables

Table 1. Table 1: An example of four headlines for the same article that were created by professional editors. In this example, ‘電動車’(Electric cars) and ‘全’(all) are represented by red letters and are not included in the 24-character headline. These tokens cannot be evaluated by 24-character headlines. The blue tokens are not included in 9- and 13-character headlines. These tokens should not be included in shorter headlines.

Article: トヨタ自動車は18日、エンジン車だけの車種を2025年ごろまでにゼロにすると発表した。…ハイブリッド車 (HV)やプラグインハイブリッド車 (PHV)、燃料電池車 (FCV)も加えた「電動車」を、すべての車種に設定する。…

On the 18th, Toyota announced that it would set the model of only engine cars to zero by about 2025.…They set “electric vehicle” which is Hybrid Vehicle (HV), Plug-in Hybrid Vehicle (PHV), and Fuel Cell Vehicle (FCV) to all models.…

Headline for print media: トヨタ、全車種に電動車　25年ごろまでに　HVやFCV含め

All Toyota models will contain electric vehicles including HV and FCV by about 2025.

Multi-length headlines for digital media:

9 chars 全車種に「電動車」 (10char-ref) “Electric cars” for all models 13 chars トヨタ、全車種に「電動車」 (13char-ref) “Electric cars” for all Toyota’s models 24 chars (26char-ref) トヨタ、エンジン車だけの車種ゼロへ　2025年ごろ Toyota sets the number of models with only engine cars to zero by about 2025.

Table 2. Table 2: ROUGE-1 precision and recall scores when comparing article and length-insensitive/sensitive headlines.

System	Reference	Precision	Recall
JAMUL Article	Paper headline	11.26	72.91
	10char-ref	5.42	85.02
	13char-ref	7.10	85.74
	26char-ref	13.26	77.38
AEG Article	AEG headline	17.74	62.03

Table 3. Table 3: ROUGE-1 precision and recall scores when comparing length-insensitive and -sensitive headlines.

System	Reference	Precision	Recall
Print headline	10char-ref	27.95	71.28
	13char-ref	37.70	74.48
26char-ref	61.78	60.56

Table 4. Table 4: A typical example for comparing length-insensitive and -sensitive headlines

Article: 米フェイスブック(FB)は1日、2017年7 9月期決算を発表し、モバイル広告の伸びなどで売上高、純利益ともに四半期として過去最高を記録した。…FB上で偽ニュースの拡散防止など、安全確保のための要員を約2万人に倍増させることを明らかにしている。

On the 1st U.S. Facebook (FB) announced financial results from July to September in 2017 and achieved a record quarterly amount of sales and net income thanks to its mobile advertising growth and other factors. …FB revealed it had doubled the number of personnel engaged in preventing fake news from spreading to about 20,000 to provide safety on FB.

Headline for print media: フェイスブック、四半期で最高益　モバイル広告好調

Facebook achieved a record quarterly profit thanks to its mobile advertising business.

Multi-length headlines for digital media:

7 chars 米FBが最高益 (10char-ref) U.S. FB achieved record profit. 12 chars 米フェイスブックが最高益 (13char-ref) U.S. Facebook achieved record profit. 26 chars (26char-ref) 米フェイスブックが最高益　偽ニュース対策で要員倍増も U.S. Facebook achieved record profit and doubled its personnel for countermeasures against fake news.

Table 5. Table 5: Difference between 26char-ref headlines and shorter headlines. P and R denote ROUGE-1 precision and recall scores, respectively.

System	Reference	P	R
26char-ref	10char-ref	30.22	81.77
26char-ref	13char-ref	43.59	90.75
First 10 chars in 26char-ref	10char-ref	44.30	29.42
First 13 chars in 26char-ref	13char-ref	67.01	44.34

Table 6. Table 6: Parameters of each encoder-decoder model.

	Seq2Seq	ConvS2S	Transformer
Num of Layer	2	8	6
Dropout Rate	0.3	0.1	0.3
Grad Clipping	[-5.0, 5.0]	[-0.1, 0.1]	-
Learning Rate	0.001	0.2	0.001
Optimizer	Adam	NAG	Adam

Table 7. Table 7: ROUGE scores of each model on the JAMUL. The specified lengths are 10, 13, and 26 characters. R-1, R-2, and R-L represent ROUGE-1, ROUGE-2, and ROUGE-L, respectively. Note that (1) to (6) in Table  8 and after that represent models (1) to (6) of this table.

	10 characters			13 characters			26 characters
Models	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
(1) Seq2Seq + LenEmb	34.62	15.02	33.54	39.31	18.14	37.07	43.65	19.73	36.03
(2) Seq2Seq + LenInit	36.14	16.88	35.15	41.39	19.55	38.83	46.30	21.33	37.79
(3) Seq2Seq + SP-token	38.01	17.16	36.72	42.03	19.83	39.56	46.62	21.47	38.29
(4) ConvS2S + SP-token	39.20	18.66	38.07	42.63	20.00	40.38	47.41	21.47	38.04
(5) ConvS2S + LC	34.93	15.52	34.02	38.80	17.21	36.68	42.59	19.34	35.41
(6) Transformer + SP-token	41.21	19.16	39.91	44.84	21.62	42.09	49.66	23.69	40.65

Table 8. Table 8: ROUGE scores of the system outputs in 10 and 13 characters evaluated by 26char-ref headlines as the references. Note that [] denote the change of the rank when we compare the rank in Table  7 ; + and - denote the rank up and rank down, respectively.

	10 characters			13 characters
Models	R-1	R-2	R-L	R-1	R-2	R-L
(1)	${21.58}_{\left[+0\right]}$	${8.73}_{\left[+1\right]}$	${19.56}_{\left[+0\right]}$	${27.47}_{\left[-1\right]}$	${11.82}_{\left[+0\right]}$	${24.20}_{\left[-1\right]}$
(2)	${22.17}_{\left[+0\right]}$	${9.14}_{\left[+1\right]}$	${20.12}_{\left[+0\right]}$	${28.93}_{\left[+1\right]}$	${12.98}_{\left[+2\right]}$	${25.35}_{\left[+1\right]}$
(3)	${22.80}_{\left[+1\right]}$	${9.51}_{\left[+1\right]}$	${20.59}_{\left[+1\right]}$	${28.78}_{\left[-1\right]}$	${12.64}_{\left[-1\right]}$	${25.14}_{\left[-1\right]}$
(4)	${22.53}_{\left[-1\right]}$	${9.04}_{\left[-2\right]}$	${20.47}_{\left[-1\right]}$	${29.23}_{\left[+0\right]}$	${12.82}_{\left[-1\right]}$	${25.58}_{\left[+0\right]}$
(5)	${21.60}_{\left[+0\right]}$	${8.55}_{\left[-1\right]}$	${19.63}_{\left[+0\right]}$	${27.63}_{\left[+1\right]}$	${11.72}_{\left[+0\right]}$	${24.34}_{\left[+1\right]}$
(6)	${24.36}_{\left[+0\right]}$	${10.32}_{\left[+0\right]}$	${21.97}_{\left[+0\right]}$	${30.84}_{\left[+0\right]}$	${13.85}_{\left[+0\right]}$	${26.98}_{\left[+0\right]}$
$\tau$	0.867	0.600	0.867	0.733	0.733	0.733

Table 9. Table 9: ROUGE scores of each model on the length-restricted JNC test set.

	10 characters			13 characters			26 characters
Models	R-1	R-2	R-L	R-1	R-2	R-L	R-1	R-2	R-L
(1)	${37.95}_{\left[+0\right]}$	${19.97}_{\left[+1\right]}$	${36.54}_{\left[+0\right]}$	${41.54}_{\left[+1\right]}$	${21.60}_{\left[+1\right]}$	${39.41}_{\left[+1\right]}$	${33.97}_{\left[+0\right]}$	${13.54}_{\left[-1\right]}$	${27.66}_{\left[+0\right]}$
(2)	${38.35}_{\left[-1\right]}$	${20.71}_{\left[+1\right]}$	${37.04}_{\left[-1\right]}$	${41.52}_{\left[-1\right]}$	${21.13}_{\left[-1\right]}$	${39.05}_{\left[-1\right]}$	${35.50}_{\left[+0\right]}$	${14.18}_{\left[+1\right]}$	${28.64}_{\left[+0\right]}$
(3)	${41.35}_{\left[+1\right]}$	${21.95}_{\left[+1\right]}$	${39.93}_{\left[+1\right]}$	${43.62}_{\left[+1\right]}$	${22.51}_{\left[+0\right]}$	${40.89}_{\left[+1\right]}$	${35.66}_{\left[+0\right]}$	${14.14}_{\left[-1\right]}$	${28.80}_{\left[+0\right]}$
(4)	${39.33}_{\left[-1\right]}$	${20.47}_{\left[-2\right]}$	${37.46}_{\left[-2\right]}$	${40.38}_{\left[-4\right]}$	${20.47}_{\left[-4\right]}$	${37.53}_{\left[-4\right]}$	${36.84}_{\left[+0\right]}$	${14.42}_{\left[+0\right]}$	${29.21}_{\left[+0\right]}$
(5)	${39.21}_{\left[+2\right]}$	${19.59}_{\left[-1\right]}$	${37.69}_{\left[+2\right]}$	${42.99}_{\left[+3\right]}$	${22.94}_{\left[+4\right]}$	${40.53}_{\left[+3\right]}$	${33.36}_{\left[+0\right]}$	${13.55}_{\left[+1\right]}$	${27.59}_{\left[+0\right]}$
(6)	${45.14}_{\left[+0\right]}$	${26.35}_{\left[+0\right]}$	${43.70}_{\left[+0\right]}$	${47.09}_{\left[+0\right]}$	${25.80}_{\left[+0\right]}$	${43.95}_{\left[+0\right]}$	${37.86}_{\left[+0\right]}$	${15.68}_{\left[+0\right]}$	${30.46}_{\left[+0\right]}$
$\tau$	0.733	0.600	0.600	0.067	-0.067	0.067	1.000	0.733	1.000

Full text

1 Introduction

The news media publish newspapers in print form and in electronic form. In the electric form, articles might be read on various types of devices using any application; thus, news media companies have an increasing need to produce multiple headlines for the same news article based on what would be most appropriate and most compelling on an array of devices. All devices and applications used for viewing articles have strict upper bounds regarding the number of characters allowed because of limitations in the space where the headline appears. The technology of automatic headline generation has the potential to contribute greatly to this domain, and the problems of news headline generation have motivated a wide range of studies (Wang et al., 2018; Chen et al., 2018; Kiyono et al., 2018; Zhou et al., 2018; Cao et al., 2018; Wang et al., 2019).

Table 1 shows sample headlines in three different lengths written by professional editors of a media company for the same news article: The length of the first headline for the digital media is restricted to 10 characters, the second to 13 characters, and the third to 26 characters. From a practical perspective, headlines must be generated under a rigid length constraint.

The first study to consider the length of system outputs in the encoder-decoder framework was Rush et al. (2015). This study controlled the length of an output sequence by reducing the score of the end-of-sentence token to $-\infty$ until the method generated the desired number of words. Subsequently, Kikuchi et al. (2016) and Fan et al. (2018) proposed mechanisms for length control; however, these studies produced summaries of 30, 50, and 75 bytes, and the studies evaluated the summaries by using the reference summaries of a single length (approximately 75 bytes long) in DUC 2004111https://duc.nist.gov/duc2004/. In addition, Takase and Okazaki (2019) proposed the mechanism for length control and evaluated their method with part of the test set which is consisted by summaries satisfying some length constraints in Annotated English Gigaword corpus (AEG) (Napoles et al., 2012).

Thus, some questions can be posed: (1) Can previous evaluation settings adequately evaluate system outputs in headline generation task? (2) What type of problem should we solve in this task according to the target length? (3) How well do systems solve the problems? In this study, we present novel corpora to investigate these research questions. The contributions of this study are threefold.

We release the Japanese News Corpus (JNC)222https://cl.asahi.com/api_data/jnc-jamul-en.html, which includes 1.93 million pairs of headlines and the lead three sentences of Japanese news articles. We expect this corpus to provide common supervision data for headline generation. 2. 2.

We build the JApanese MUlti-Length Headline Corpus (JAMUL)22footnotemark: 2 for the evaluation of headlines of different lengths. In this novel dataset, each news article is associated with multiple headlines of three different lengths. 3. 3.

We report new findings for the JAMUL; for example, although the longer reference seems to be able to evaluate the short system output, we also found a problem with this evaluation setting. Additionally, we clarified that the existing methods could not capture what must be changed according to the specified length.

2 JNC and JAMUL

2.1 Headlines Composed by a Media Company

Before describing the JNC and the JAMUL in detail, we explain the process where a Japanese media company composes headlines for a news article. We describe the process of The Asahi Shimbun Company, a Japanese newspaper company. First, reporters write an article and submit it to the editorial department. The editorial department picks up some articles to publish in the newspaper and writes a headline for the article dedicated to print media. We call these headlines print headlines or length-insensitive headlines hereafter.

In addition to print headlines, digital media editors, who are typically not the same editors for print, pick up the articles they want to distribute on digital media from the articles dedicated to digital media and compose three different headlines. The first headline for the digital signage and audio media has a limit of up to 10 characters. This type of headline is appended to the beginning of a concise summary of the article so that readers can understand the news at first glance. The second type of headline is produced for mobile phones with small LCDs and small areas on the website sidebar (e.g., the access ranking); the upper limit of the number of characters is 13. The third type of headline is produced for desktop computer news websites, and the upper limit of the number of characters is 26. This limit is derived from the layout of the news site. We refer to the three types of headlines as 10char-ref, 13char-ref, and 26char-ref (refer to Table 1 for examples). We collectively call these headlines length-sensitive headlines.

Table 1 presents an example of headlines written for an article by the professional editors. We extract the JNC and the JAMUL from the news production process of The Asahi Shimbun Company; therefore, they can be considered representative of contemporary editorial practice.

2.2 JNC

The JNC is a collection of 1,932,398 pairs of the three lead sentences22footnotemark: 2 of articles and their print headlines published by The Asahi Shimbun Company from 2007 to 2016. Figure 1 (a) depicts the distribution of the headline lengths in the JNC. The headline lengths in the JNC are diverse because almost all headlines are not restricted by length limitations.

The JNC is useful for training headline generation models because it has many training instances. Furthermore, the corpus is suitable for training a model for variable-length headline generation because of the variety of the headline lengths.

2.3 JAMUL

The JAMUL is a corpus containing 1,489 full-text news articles and their length-insensitive headlines for print media and length-sensitive headlines of 10 characters, 13 characters, and 26 characters for digital media. We just extract these articles and headlines from the company database. All the articles and headlines were published by The Asahi Shimbun Company between September 2017 and March 2018. The volume of the news articles may be insufficient for training a headline generation model. However, as Figure 1 (b) shows, the JAMUL includes length-sensitive headlines that strictly preserve the length requirements. This novel characteristic of the JAMUL is suitable for the test set to adapt headline generation to business practice. In this paper, we use the first three sentences of the article as input because of fitting in the JNC corpus. But we publish the full edition of the article. No overlap of articles between the JNC and the JAMUL was confirmed.

2.4 Comparing of Headlines with Article Bodies

What type of operation which includes extractive and abstractive operations did the editors perform to create length-sensitive and length-insensitive headlines in the JAMUL? To clarify this question, we analyzed the proportions of the number of extractive and abstractive operations. Specifically, we reported ROUGE-1 precision and recall scores (Lin, 2004) in Table 2, assuming that articles are “system” summaries and that 10char-ref, 13char-ref, and 26char-ref headlines are “reference” summaries. Notably, we removed blank spaces, which were the most common token in longer headlines. The relatively high recall score indicates that the operations most often required to generate headlines are extractive, and the abstractive operation is about 15–27% of the total. In addition, we explored the proportions of each operation in AEG333We obtained this dataset by applying the pre-processing script at https://github.com/facebookarchive/NAMAS, which has been used in many studies for English headline generation tasks. From the relatively high recall score, we observed that Japanese headlines tend to be more extractive.

2.5 Comparing of Length-sensitive Headlines with Print headlines

How similar are the headlines used for training (length-insensitive) and for evaluation (length-sensitive)? We estimated the appropriateness of length-insensitive headlines as a “seed” for producing length-sensitive headlines. More concretely, we report ROUGE-1 precision and recall scores in Table 3, assuming that length-insensitive headlines are “system” summaries, and that 10char-ref, 13char-ref and 26char-ref headlines are “reference” summaries. The relatively high recall scores indicate that the training and evaluation data were not very distant. Additionally, we found that the editors used a moderate number of words that did not appear in print headlines when composing length-sensitive headlines. Table 4 is an example of the typical differences between the length-insensitive and length-sensitive headlines. Comparing the 26-character headline with the print headline, the choices of contents are different; for example, the print headline reports the reason for the record profit, but the 26-character headline describes the topic regarding the increasing number of personnel. Next, comparing the 7-character (10char-ref) headline with the print headline, we observe that the choices of words are different; the print headline uses “Facebook,” which is changed to “FB” in the 7-character headline.

2.6 Comparing of Length-sensitive Headlines

How similar is the composition of headlines for a news article of different lengths? How good are 26char-ref headlines as “seeds” for generating 10char-ref or 13char-ref headlines? Is the simple strategy of trimming 26char-ref headlines to 10 or 13 characters sufficient? To answer these questions, we computed word-level precision and recall scores, assuming that 26char-ref headlines are “system” summaries and that 10char-ref and 13char-ref headlines are “reference” summaries.

The first and second rows of Table 5 represent the situation when we used 26char-ref headlines as they are, and without preserving the length constraint. Although this setting was unrealistic, we could estimate how similar the contents were among the length-sensitive headlines. The high recall scores indicate that 26char-ref headlines mostly cover the words included in the 10char-ref and 13char-ref headlines. From this result, we consider the contents among length-sensitive headlines are comparably similar. The third and fourth rows of Table 5 correspond to the strategy where we generated headlines in 10 and 13 characters from the first 10 and 13 characters of the 26char-ref headlines. This strategy achieved moderate success for generating headlines in 13 characters, but did not work well for headlines in 10 characters. In other words, we observed large differences between the 10char-ref and the first 10 characters of the 26char-ref headlines. Therefore, we need to re-generate the shorter headline, as well as the longest one.

In sum, we found similarities in headlines of different lengths in the JAMUL. However, the simple strategy of trimming a longer headline into a shorter headline is insufficient. Table 1 is an example of the typical differences among length-sensitive headlines. There is a little overlap between longer and shorter headlines because the 9- and 13-character headlines extract the shorter phrases which have the nearly same meaning as the 24-character headline. Focusing on “車種” (models), the words are in the latter half of the 24-character headline, and we could confirm that important keywords are not always included at the beginning of the headlines.

3 Comparing of Headline Generation Methods on the JAMUL

In this section, we explore a question about evaluation: How reliable is the two conventional evaluation method; the first is the method that uses a single length summary for measuring the quality of summaries of different lengths; the second is the method that uses the specified length headlines which are extracted from the dataset constructed by length-insensitive headlines, for measuring the quality of each length summary. To answer this question, we generated multiple summaries of different lengths by using the existing methods, and measured the correlation between the performance values computed by the conventional evaluation methods and those computed on the JAMUL.

3.1 Headline-generation Methods with the Mechanism to Control the Output Length

In this study, we explored four methods for headline generation that can control the output length. The first two methods, LenEmb and LenInit, were proposed by Kikuchi et al. (2016).

LenEmb provides the decoder with output length information in the form of the length embedding. LenInit controls the output length by multiplying the initial state of the decoder’s memory cell of long short-term memory (Hochreiter and Schmidhuber, 1997) by the desired length.

Fan et al. (2018) also proposed a length-controllable method for a convolutional sequence-to-sequence (ConvS2S) model (Gehring et al., 2017). Their method added special tokens indicating the range of the output length at the beginning of an input sequence. In our experiment, we used a special token to specify an output length444Fan et al. (2018) also included special tokens for entities, but we did not use them in the experiments. and called this method SP-token.

We also considered the LC method (Liu et al., 2018), which extends ConvS2S and multiplies the initial state of the residual connection (He et al., 2016) by the desired number of output tokens. In the experiment, we set the desired number of characters instead of that of tokens.

In addition to these four methods, we combined SP-token not only on ConvS2S but also on Seq2Seq and Transformer (Vaswani et al., 2017). Eventually, we examined six combinations in total: (1) Seq2Seq $+$ LenEmb, (2) Seq2Seq $+$ LenInit, (3) Seq2Seq $+$ SP-token, (4) ConvS2S $+$ SP-token, (5) ConvS2S $+$ LC, and (6) Transformer $+$ SP-token.

3.2 Datasets and the Evaluation Protocol

We trained the six methods for headline generation on the JNC. We removed instances that were duplicated or unsuitable for training a headline generation model555The filtering script is available at:

https://github.com/asahi-research/Gingo. The filtering step obtained 1,554,558 pairs of newspaper articles and headlines. We randomly selected 98% of the instances (1,523,468 pairs) as a training set, selected 1% of the instances (15,545 pairs) as a validation set, and used the remainder (15,545 pairs) as a test set. As for the JNC test set, we additionally extracted the length-restricted test set to evaluate the length control methods. Specifically, we extracted 303, 1,436, and 4,289 headlines which are 8-10, 11-13, and 24-26 characters long, respectively, and corresponding articles from the JNC, similar to the 10char-ref, 13char-ref, and 26char-ref in the JAMUL. We call them the length-restricted JNC test set. We also filtered the JAMUL by setting lower bounds for the length of the headlines66footnotemark: 6. We set each lower bound to 8 for 10char-ref, to 11 for 13char-ref, and to 24 for 26char-ref. Finally, we achieved 1,288 instances for the JAMUL test set.

We used SentencePiece666https://github.com/google/sentencepiece (Kudo and Richardson, 2018) for tokenization. We set the merge operation to 8,000. Finally, we obtained 10,868 tokens for the source side and 9,556 tokens for the target side. When training a model, we set the length of each reference headline to the model. When generating headlines in the evaluation, we set the output lengths to 10, 13, and 26 characters; each output was evaluated by the reference that had the same length in the JAMUL and JNC test sets. To generate summaries, we follow standard practice in disallowing repetition of the same trigram (Paulus et al., 2018). We evaluated all models by using three variants of ROUGE recall metric777We used MeCab (Kudo et al., 2004) to tokenize the system outputs.: ROUGE-1, ROUGE-2, and ROUGE-L. Headlines exceeding the length limits were trimmed for a fair evaluation.

3.3 Implementation

We employed OpenNMT888https://github.com/OpenNMT/OpenNMT-py (Klein et al., 2017) for Seq2Seq, and fairseq999https://github.com/pytorch/fairseq (Ott et al., 2019) for ConvS2S and Transformer. We extended the implementations to realize LenEmb, LenInit, and LC. We set the dimensions for the token and length embeddings to 512, those for hidden states to 512, and the beam width to 5. These parameters are common in all the models. Table 6 summarizes other parameters specific to each sequence-to-sequence model. We used Nesterov’s accelerated gradient method (NAG) (Sutskever et al., 2013) with a momentum of 0.99 in ConvS2S. In Transformer, we set the number of attention heads to 8, the dimensions for the feed-forward network to 2,048, Adam’s $\beta$ to 0.98, the warm-up steps to 4,000, and label smoothing to 0.1.

3.4 Evaluation of Multi-length Headlines Generated by Methods on the JAMUL

Table 7 presents ROUGE scores of each method on the JAMUL test set. Transformer + SP-token was the clear winner in all length and evaluation metrics. Additionally, the three methods with SP-token outperformed the others.

What if we do not have multiple headlines of different lengths to evaluate the methods? To answer this question, we followed the evaluation setup of the previous studies on DUC 2004: The reference summaries of 75 bytes were used even when evaluating summaries of 30 and 50 bytes. Table 8 reports ROUGE scores for the system outputs in 10 and 13 characters evaluated based on the 26char-ref headlines. This evaluation setup reduced the performance differences between the methods. Although Transformer + SP-token remained the clear winner, the ranking in ROUGE scores of the other methods are flipped. We also computed rank correlation coefficients (Kendall’s $\tau$) to assess the discrepancy in the ranking among the methods presented in Table 7 and Table 8. The last row of Table 8 reveals that the rank correlation is not perfect (lower than one) but moderate. We understand that $\tau$ is maintained high to some extent because of two reasons: (1) Most of the swaps occur in adjacent ranks, and (2) there are not many samples of the method in this analysis. Then, we focus on the samples flipped in this evaluation setup, as shown in Table 8. We observed that the rankings are swapped even if there is a difference of 1.0 or more (these differences are well observed in state-of-the-art competition) in Table 7. For example, we can show this issue between (3) and (4) in 10 characters of R-1 score, (2) and (4) in 10 characters of R-2 score, and (3) and (4) in 10 characters setting of R-L score in Table 7.

What if we do not have strict length headlines to evaluate the methods? To answer this question, we followed the evaluation setup of previous studies : The reference summaries of each target length were extracted from the JNC test set. Table 9 reports ROUGE scores for the system outputs in 10, 13, and 26 characters evaluated in the length-restricted JNC test set. In this evaluation setup, Transformer + SP-token remained the clear winner, but the performances of the other methods were flipped. We also calculated $\tau$ in this case. The last row of Table 9 reveals that the 13 character setting considerably lost correlation with the JAMUL evaluation result. We guess this inconsistency is brought about by using the different content test sets among each length setting. Depending on the target length which is specified by the evaluation setting, this evaluation protocol might not evaluate the methods adequately.

4 Analysis

4.1 Performance of Word Selection According to the Output Length

How well do the existing methods change the word selection depending on the output length? As shown in the first and second rows of Table 5, the 10char-ref and 13char-ref headlines contain words that are not included in the 26char-ref headlines. In other words, the selection of words in the generated headline should be changed in response to the length restriction. To confirm this question, we computed ROUGE-1 recall scores for the system outputs generated by each method, assuming that the groups of the words included in the 10char-ref or 13char-ref but not in the 26char-ref headlines are the “reference” summaries. For instance, the red words in Table 1 are the “reference” summaries in this experiment.

We report this result in Figure 2. The low recall score indicates that each system cannot select the words tailored to the length constraints. However, (4) ConvS2S + SP-token shows the highest performance. Taking this into account, Transformer + SP-token improves the important words in all lengths setting, but ConvS2S + SP-token may improve the change of words according to the target length.

4.2 Performance of Managing Extractive and Abstractive Tasks

In Table 2, we reported the proportion of the number of extractive and abstractive operations in the JAMUL. We analyze how the existing methods can reflect extractive and abstractive operations in generating summaries.

First, to observe extractive operations, we computed ROUGE-1 recall scores for the system outputs generated by each system, measuring the number of overlapping words between an article as “system” summaries and 10char-ref, 13char-ref, and 26char-ref headlines as “reference” summaries. The group of bars which contained ext in its name in Figure 3 reports the result. The relatively high recall score indicates that the length control method succeeds in managing extractive operations.

Next, we examine whether the length control methods could perform abstractive operations. We adopted the words included in 10char-ref, 13char-ref, or 26char-ref headlines but not included in an article as “reference” summaries, and computed the ROUGE-1 recall scores for the system outputs. The group of bars which contained abs in its name in Figure 3 reports the result. Regarding the outputs targeting 26 characters, the recall scores of around 30 point imply that each model can manage abstractive operations to some extent. In contrast, the low recall scores for the outputs targeting 10 and 13 characters revealed that all length control methods could not perform well on abstractive operations under the severe length constraint.

4.3 How Do Length Control Mechanisms Work?

We wondered whether a method that could control the output length would produce similar headlines even for different lengths for the same news article. To confirm this suspicion, we reported ROUGE-1 recall scores in Figure 4 with three different configurations: (a) evaluating the first 13 characters of headlines generated to be 26 characters long on 13char-ref headlines (blue); (b) evaluating headlines generated to be 13 characters long on 13char-ref headlines (pink); and setting (c) is the same as setting (a) but evaluated on the headline generated to be 13 characters long (green).

Setting (a) corresponds to the strategy where we trimmed headlines of different lengths to 13 characters long. This setting was worse than setting (b), where a method tailored headlines to the desired length. However, the difference in ROUGE scores between (a) and (b) was not very large, indicating that the existing methods do not drastically change the content for 13 characters long and 26 characters long. This tendency was also verified by setting (c), which assessed how much the first 13 characters of headlines generated to be 26 characters long covered the content of those generated to be 13 characters long. These facts suggest that we should explore in further research a method not only trained by generic supervision data (print headlines) but also tuned for the desired length.

5 Related Work

Rush et al. (2015) created the first approach to neural abstractive summarization. They generated a headline from the first sentence of a news article in the AEG (Napoles et al., 2012), which contains an enormous number of pairs of headlines and articles. After their study, a number of researchers addressed this task: For example, Chopra et al. (2016) used the encoder-decoder framework (Sutskever et al., 2014; Bahdanau et al., 2015) and Nallapati et al. (2016) incorporated additional features into the model, such as parts-of-speech tags and named entities. Suzuki and Nagata (2017) proposed word-frequency estimation to reduce the repeated phrases being generated. Zhou et al. (2017) proposed a gating mechanism (sGate) to ensure that important information is selected at each decoding step.

Furthermore, attempts to control the output length in neural abstractive summarization have been gradually increasing. Shi et al. (2016) reported that hidden states in recurrent neural networks in the encoder-decoder framework could implicitly model the length of the output sequences. Kikuchi et al. (2016) was the first to propose the idea of controlling the output length in the encoder-decoder framework. Their approach inserts length information for the output length into the decoder. Additionally, Fan et al. (2018) reported that output lengths could be controlled by embedding special tokens given to an input sequence. These two studies used DUC 2004 (Over et al., 2007), which comprises only 75-byte summaries, to evaluate the outputs in multiple lengths. Liu et al. (2018) also proposed a method for controlling the number of output tokens in the ConvS2S model. In Transformer (Vaswani et al., 2017), Takase and Okazaki (2019) proposed two length control methods by extending positional embedding. They additionally evaluated the system outputs by using the reconstructed test set of AEG which consists of the fixed length headlines. Makino et al. (2019) proposed a global optimization method under a length constraint.

Sun et al. (2019) examined how to compare summarizers by considering the length bias of generated summaries in the test set that includes various length summaries. However, no previous work built a dataset for evaluating headlines of multiple lengths or reported an in-depth perspective on this task during the process of new production in the real world. However, a single length reference that could appropriately evaluate multiple length summaries in multiple document summarization was reported Shapira et al. (2018). In that study, the authors confirmed the correlation coefficient of ROUGE scores between the scores using a single length reference and multiple (gold) length references in the evaluation. The present research differed in that we examined what kind of problems occurred and studied headline generation domain, which requires stricter keyword selection.

6 Conclusion

In this paper, we presented two new corpora: The JNC contains a large number of pairs of news articles and their headlines, and the JAMUL includes headlines of three different lengths (10, 13, and 26 characters long) written by professional editors. This study is the first to analyze the characteristics of multiple headlines of different lengths, and to evaluate existing approaches for length control based on the reference headlines composed for different lengths. We found that Transformer model with a special length token (SP-token) outperformed the other methods on the JAMUL. Additionally, although we confirmed that single length (the longest) references could adequately evaluate multiple length system outputs, the rankings were swapped even if there is a difference of 1.0 or more in the ROUGE scores. We also confirmed the length-restricted test set, which was extracted from length-insensitive headlines and corresponding articles, could not adequately evaluate the multiple length system outputs depending on the specified length. In the analysis, the existing methods could not take into account the word selection according to length constraint. We also found it difficult to evaluate methods to controlling output length, because headlines of different lengths are written based on different goals, and because the training data does not necessarily reflect the goal of the headlines of a specific length. In the future, we plan to explore an approach to adapt a model trained on print headlines to those which dedicated to a different length.