Query-focused Sentence Compression in Linear Time

TL;DR

This paper presents a linear-time, query-focused sentence compression method that efficiently produces constrained shortenings, significantly faster than ILP-based approaches, improving user interface responsiveness without needing complex solvers or GPUs.

Contribution

Introduces a novel linear-time transition-based sentence compression technique for query-focused applications, outperforming ILP methods in speed and accuracy.

Findings

Achieves 11X speedup over ILP baselines

Better reconstructs gold constrained shortenings

Does not require ILP solver or GPU hardware

Abstract

Search applications often display shortened sentences which must contain certain query terms and must fit within the space constraints of a user interface. This work introduces a new transition-based sentence compression technique developed for such settings. Our query-focused method constructs length and lexically constrained compressions in linear time, by growing a subgraph in the dependency parse of a sentence. This theoretically efficient approach achieves an 11X empirical speedup over baseline ILP methods, while better reconstructing gold constrained shortenings. Such speedups help query-focused applications, because users are measurably hindered by interface lags. Additionally, our technique does not require an ILP solver or a GPU.

Figures

Tables

Table 1. Table 1: Our vertex addition technique (§ 3 ) constructs constrained compressions in linear time. Prior work (§ 2 ) has higher computational complexity ( ILP ) or does not respect hard constraints (LSTM tagger).

Approach	Complexity	Constrained
ilp	exponential	yes
LSTM tagger	linear	no
vertex addition	linear	yes

Table 2. Table 2: Test results for constrained compression. ∗ Latency is the geometric mean of observed runtimes (in milliseconds per sentence). vertex addition LR achieves the highest F1, and also runs 10.73 times faster than the ilp . Differences between all scores for vertex addition LR and ilp are significant ( p < .01 ) 𝑝 .01 (p<.01) .

Approach	F1	SLOR	∗Latency
random (lower bound)	0.653	0.377	0.5
ablated (edge only)	0.827	0.669	3.7
vertex additionNN	0.873	0.728	2929.1 (CPU)
ilp	0.852	0.756	44.0
vertex additionLR	0.881	0.745	4.1

Table 3. Table 3: Hyperparameters for vertex addition NN

Learning rate	0.025
Embedding dim.	315
Weight decay	1.88 $\times {10}^{-9}$
Hidden dim.	158
Batch size	135

Table 4. Table 4: Mean test time compression ratios for all techniques. We also show mean ratios for gold compressions C g subscript 𝐶 𝑔 C_{g} across the train and test sets.

random	0.405
ilp	0.408
ablated	0.387
vertex additionLR	0.403
vertex additionNN	0.405
$C_g$ Train	0.384
$C_g$ Test	0.413

Equations

$$\text{SLOR}=\frac{\text{log}{P}_m(\xi )-\text{log}{P}_u(\xi )}{|\xi |}$$

Full text

1 Introduction

Traditional study of extractive sentence compression seeks to create short, readable, single-sentence summaries which retain the most “important” information from source sentences. But search user interfaces often require compressions which must include a user’s query terms and must not exceed some maximum length, permitted by screen space. Figure 1 shows an example.

This study examines the English-language compression problem with such length and lexical requirements. In our constrained compression setting, a source sentence $S$ is shortened to a compression $C$ which (1) must include all tokens in a set of query terms $Q$ and (2) must be no longer than a maximum budgeted character length, $b\in\mathbb{Z}^{+}$. Formally, constrained compression maps $(S,Q,b)\rightarrow C$, such that $C$ respects $Q$ and $b$. We describe this task as query-focused compression because $Q$ places a hard requirement on words from $S$ which must be included in $C$.

Existing techniques are poorly suited to constrained compression. While methods based on integer linear programming (ILP) can trivially accommodate such length and lexical restrictions Clarke and Lapata (2008); Filippova and Altun (2013); Wang et al. (2017), these approaches rely on slow third-party solvers to optimize an NP-hard integer linear programming objective, causing user wait time. An alternative LSTM tagging approach Filippova et al. (2015) does not allow practitioners to specify length or lexical constraints, and requires an expensive graphics processing unit (GPU) to achieve low runtime latency (access to GPUs is a barrier in fields like social science and journalism). These deficits prevent application of existing compression techniques in search user interfaces Marchionini (2006); Hearst (2009), where length, lexical and latency requirements are paramount. We thus present a new stateful method for query-focused compression.

Our approach is theoretically and empirically faster than ILP-based techniques, and more accurately reconstructs gold standard compressions.

2 Related work

Extractive compression shortens a sentence by removing tokens, typically for summarization Knight and Marcu (2000); Clarke and Lapata (2008); Filippova et al. (2015); Wang et al. (2017).111Some methods compress via generation instead of deletion Rush et al. (2015); Mallinson et al. (2018). Our extractive method is intended for practical, interpretable and trustworthy search systems Chuang et al. (2012). Users might not trust abstractive summaries Zhang and Cranshaw (2018), particularly in cases with semantic error. To our knowledge, this work is the first to consider extractive compression under hard length and lexical constraints.

We compare our vertex addition approach to ILP-based compression methods Clarke and Lapata (2008); Filippova and Altun (2013); Wang et al. (2017), which shorten sentences using an integer linear programming objective. ilp methods can easily accommodate lexical and budget restrictions via additional optimization constraints, but require worst-case exponential computation.222ILPs are exponential in $|V|$ when selecting tokens Clarke and Lapata (2008) and exponential in $|E|$ when selecting edges Filippova et al. (2015).

Finally, compression methods based on LSTM taggers Filippova et al. (2015) cannot currently enforce lexical or length requirements. Future work might address this limitation by applying and modifying constrained generation techniques Kikuchi et al. (2016); Post and Vilar (2018); Gehrmann et al. (2018).

3 Compression via vertex addition

We present a new transition-based method for shortening sentences under lexical and length constraints, inspired by similar approaches in transition-based parsing Nivre (2003). We describe our technique as vertex addition because it constructs a shortening by growing a (possibly disconnected) subgraph in the dependency parse of a sentence, one vertex at a time. This approach can construct constrained compressions with a linear algorithm, leading to 11x lower latency than ILP techniques (§4). To our knowledge, our method is also the first to construct compressions by adding vertexes rather than pruning subtrees in a parse Knight and Marcu (2000); Almeida and Martins (2013); Filippova and Alfonseca (2015). We assume a boolean relevance model: $S$ must contain $Q$. We leave more sophisticated relevance models for future work.

3.1 Formal description

vertex addition builds a compression by maintaining a state $(C_{i},P_{i})$ where $C_{i}\subseteq S$ is a set of added candidates, $P_{i}\subseteq S$ is a priority queue of vertexes, and $i$ indexes a timestep during compression. Figure 2 shows a step-by-step example.

During initialization, we set $C_{0}\leftarrow Q$ and $P_{0}\leftarrow S\setminus Q$. Then, at each timestep, we pop some candidate $v_{i}=h(P_{i})$ from the head of $P_{i}$ and evaluate $v_{i}$ for inclusion in $C_{i}$. (Neighbors of $C_{i}$ in $P_{i}$ get higher priority than non-neighbors; we break ties in left-to-right order, by sentence position). If we accept $v_{i}$, then $C_{i+1}\leftarrow C_{i}\cup v_{i}$; if not, $C_{i+1}\leftarrow C_{i}$. (Acceptance decisions are detailed in §4.3.) We continue adding vertexes to $C$ until either $P_{i}$ is empty or $C_{i}$ is $b$ characters long.333We linearize $C$ by left-to-right vertex position in $S$, common for compression in English Filippova and Altun (2013). The appendix includes a formal algorithm.

vertex addition is linear in the token length of $S$ because we pop and evaluate some vertex from $P_{i}$ at each timestep, after $P_{0}\leftarrow S\setminus Q$. Additionally, because (1) we never accept $v_{i}$ if the length of $C_{i}\cup v_{i}$ is more than $b$, and (2) we set $C_{0}\leftarrow Q$, our method respects $Q$ and $b$.

4 Evaluation

We observe the latency, readability and token-level F1 score of vertex addition, using a standard dataset Filippova and Altun (2013). We compare our method to an ilp baseline (§2) because ILP methods are the only known technique for constrained compression. All methods have similar compression ratios (shown in appendix), a well-known evaluation requirement Napoles et al. (2011). We evaluate the significance of differences between vertex additionLR and the ilp with bootstrap sampling Berg-Kirkpatrick et al. (2012). All differences are significant $(p<.01)$.

4.1 Constrained compression experiment

In order to evaluate different approaches to constrained compression, we require a dataset of sentences, constraints and known-good shortenings, which respect the constraints. This means we need tuples $(S,Q,b,C_{g})$, where $C_{g}$ is a known-good compression of $S$ which respects $Q$ and $b$ (§1).

To support large-scale automatic evaluation, we reinterpret a standard compression corpus Filippova and Altun (2013) as a collection of input sentences and constrained compressions. The original dataset contains pairs of sentences $S$ and compressions $C_{g}$, generated using news headlines. For our experiment, we set $b$ equal to the character length of the gold compression $C_{g}$. We then sample a small number of nouns4441 to 3 nouns; cardinality chosen uniformly at random. from $C_{g}$ to form a query set $Q$, approximating both the observed number of tokens and observed parts of speech in real-world search Jansen et al. (2000); Barr et al. (2008). Sampled $Q$ include reasonable queries like “police, Syracuse”, “NHS” and “Hughes, manager, QPR”.

By sampling queries and defining budgets in this manner, we create 198,570 training tuples and 9949 test tuples, each of the form $(S,Q,b,C_{g})$. Filippova and Altun (2013) define the train/test split. We re-tokenize, parse and tag with CoreNLP v3.8.0 Manning et al. (2014). We reserve 25,000 training tuples as a validation set.

4.2 Model: ilp

We compare our system to a baseline ilp method, presented in Filippova and Altun (2013). This approach represents each edge in a syntax tree with a vector of real-valued features, then learns feature weights using a structured perceptron trained on a corpus of $(S,C_{g})$ pairs.555Another ILP Wang et al. (2017) sets weights using a LSTM, achieving similar in-domain performance. This method requires a multi-stage computational process (i.e. run LSTM then ILP) that is poorly-suited to query-focused settings, where low latency is crucial. Learned weights are used to compute a global compression objective, subject to structural constraints which ensure $C$ is a valid tree. This baseline can easily perform constrained compression: at test time, we add optimization constraints specifying that $C$ must include $Q$, and not exceed length $b$.

To our knowledge, a public implementation of this method does not exist. We reimplement from scratch using Gurobi Optimization (2018), achieving a test-time, token-level F1 score of 0.76 on the unconstrained compression task, lower than the result (F1 = 84.3) reported by the original authors. There are some important differences between our reimplementation and original approach (described in detail in the appendix). Since vertex addition requires $Q$ and $b$, we can only compare it to the ILP on the constrained (rather than traditional, unconstrained) compression task.

4.3 Models: vertex addition

Vertex addition accepts or rejects some candidate vertex $v_{i}$ at each timestep $i$. We learn such decisions $y_{i}\in\{0,1\}$ using a corpus of tuples $(S,Q,b,C_{g})$ (§4.1). Given such a tuple, we can always execute an oracle path shortening $S$ to $C_{g}$ by first initializing vertex addition and then, at each timestep: (1) choosing $v_{i}=h(P_{i})$ and (2) adding $v_{i}$ to $C_{i}$ iff $v_{i}\in C_{g}$. We set $y_{i}=1$ if $v_{i}\in C_{g}$; we set $y_{i}=0$ if $v_{i}\notin C_{g}$. We then use decisions from oracle paths to train two models of inclusion decisions, ${p(y_{i}=1|v_{i},C_{i},P_{i},S)}$. At test time, we accept $v_{i}$ if $p(y_{i}>.5)$.

Model One. Our vertex additionNN model broadly follows neural approaches to transition-based parsing (e.g. Chen and Manning (2014)): we predict $y_{i}$ using a LSTM classifier with a standard max-pooling architecture Conneau et al. (2017), implemented with a common neural framework Gardner et al. (2017). Our classifier maintains four vocabulary embeddings matrixes, corresponding to the four disjoint subsets $C_{i}\cup\neg C_{i}\cup P_{i}\cup\{v_{i}\}=V$. Each LSTM input vector $x_{t}$ comes from the appropriate embedding for $v_{t}\in V$, depending on the state of the compression system at timestep $i$. The appendix details network tuning and optimization.

Model Two. Our vertex additionLR model uses binary logistic regression,666We implement with Python 3 using scikit-learn Pedregosa et al. (2011). We tune the inverse regularization constant to $c=10$ via grid search over powers of ten, to optimize validation set F1. with 3 classes of features.

Edge features describe the properties of the edge $(u,v_{i})$ between $v_{i}\in P_{i}$ and $u\in C_{i}$. We use the edge-based feature function from Filippova and Altun (2013), described in detail in the appendix. This allows us to compare the performance of a vertex addition method based on local decisions with an ILP method that optimizes a global objective (§4.5), using the same feature set.

Stateful features represent the relationship between $v_{i}$ and the compression $C_{i}$ at timestep $i$. Stateful features include information such as the position of $v_{i}$ in the sentence, relative to the right-most and left-most vertex in $C_{i}$, as well as history-based information such as the fraction of the character budget used so far. Such features allow the model to reason about which sort of $v_{i}$ should be added, given $Q$, $S$ and $C_{i}$.

Interaction features are formed by crossing all stateful features with the type of the dependency edge governing $v_{i}$, as well as with indicators identifying if $u$ governs $v_{i}$, if $v_{i}$ governs $u$ or if there is no edge $(u,v_{i})$ in the parse.

4.4 Metrics: F1, Latency and SLOR

We measure the token-level F1 score of each compression method against gold compressions in the test set. F1 is the standard automatic evaluation metric for extractive compression Filippova et al. (2015); Klerke et al. (2016); Wang et al. (2017).

In addition to measuring F1, researchers often evaluate compression systems with human importance and readability judgements Knight and Marcu (2000); Filippova et al. (2015). In our setting $Q$ determines the “important” information from $S$, so importance evaluations are inappropriate. To check readability, we use the automated readability metric SLOR Lau et al. (2015), which correlates with human judgements Kann et al. (2018).

We evaluate theoretical gains from vertex addition (Table 1) by measuring empirical latency. For each compression method, we sample and compress $N=300,000$ sentences, and record the runtime (in milliseconds per sentence). We observe that runtimes are distributed log-normally (Figure 3), and we thus summarize each sample using the geometric mean. ilp and vertex additionLR share edge feature extraction code to support to fair comparison. We test vertex additionNN using a CPU: the method is too slow for use in search applications in areas without access to specialized hardware (Table 2). The appendix further details latency and SLOR experiments.

4.5 Analysis: ablated & random

For comparison, we implement an ablated vertex addition method, which learns inclusion decisions using only edge features from Filippova and Altun (2013). ablated has a lower F1 score than ilp, which uses the same edge-level information to optimize a global objective: adding stateful and interaction features (i.e. vertex additionLR) improves F1 score. Nonetheless, strong performance from ablated hints that edge-level information alone (e.g. dependency type) can mostly guide acceptance decisions.

We also evaluate a random baseline, which accepts each $v_{i}$ randomly in proportion to $p(y_{i}=1)$ across training data. random achieves reasonable F1 because (1) $C_{0}=Q\in C_{g}$ and (2) F1 correlates with compression rate Napoles et al. (2011), and $b$ is set to the length of $C_{g}$.

5 Future work: practical compression

This work presents a new method for fast query-focused sentence compression, motivated by the need for query-biased snippets in search engines Tombros and Sanderson (1998); Marchionini (2006). While our approach shows promise in simulated experiments, we expect that further work will be required before the method can be employed for practical, user-facing search.

To begin, both our technique and our evaluation ignore the conventions of search user interfaces, which typically display missing words using ellipses. This convention is important, because it allows snippet systems to transparently show users which words have been removed from a sentence. However, we observe that some well-formed compressions are difficult to read when displayed in this format. For instance the sentence “Aristide quickly fled Haiti in September 1991” can be shortened to the well-formed compression “Aristide fled in 1991.” But this compression does not read fluidly when using ellipses (“Aristide…fled…in…1991”). Human experiments aimed at enumerating the desirable and undesirable properties of compressions displayed in ellipse format (e.g. compressions should minimize number of ellipses?) could help guide user-focused snippet algorithms in future work.

Our method also assumes access to a reliable, dependency parse, and ignores any latency penalties incurred from parsing. In practical settings, both assumptions are unreasonable. Like other NLP tools, dependency parsers often perform poorly on out-of-domain text Bamman (2017), and users looking to quickly investigate a new corpus might not wish to wait for a parser. Faster approaches based on low-latency part-of-speech tagging, or more cautious approaches based on syntactic uncertainty Keith et al. (2018), each offer exciting possibilities for additional research.

Our approach also assumes that a user already knows a reasonable $b$ and reasonable $Q$ for a given sentence $S$.777Recall that we simulate $b$ and $Q$ based on the well-formed shortening $C_{g}$, see §4.1. However, in some cases, there is no well-formed shortening of which respects the requirements. For instance, if $Q$=“Kennedy” and $b$=15 there is no reasonable shortening for the toy sentence “Kennedy kept running”, because the compressions “Kennedy kept” and “Kennedy running” are not well-formed. We look forward to investigating which $(Q,S,b)$ triples will never return well-formed compressions in later work.

Finally, some shortened sentences will modify the meaning of a sentence, but we ignore this important complication in this initial study. In the future, we hope to apply ongoing research into textual entailment Bowman et al. (2015); Pavlick and Callison-Burch (2016); McCoy and Linzen (2018) to develop semantically-informed approaches to the task.

6 Conclusion

We introduce a query-focused vertex additionLR method for search user interfaces, with much lower theoretical complexity (and empirical runtimes) than baseline techniques. In search applications, such gains are non-trivial: real users are measurably hindered by interface lags Nielsen (1993); Liu and Heer (2014). We hope that our fast, query-focused method better enables snippet creation at the “pace of human thought” Heer and Shneiderman (2012).

Acknowledgments

Thanks to Javier Burroni and Nick Eubank for suggesting ways to optimize and measure performance of Python code. Thanks to Jeffrey Flanigan, Katie Keith and the UMass NLP reading group for feedback. This work was partially supported by IIS-1814955.

Appendix A Appendix

A.1 Algorithm

We formally present the vertex addition compression algorithm, using notation defined in §3.1. $\ell$ linearizes a vertex set, based on left-to-right position in $S$. $|P|$ indicates the number of tokens in the priority queue.

A.2 Neural network tuning and optimization

We learn network parameters for vertex additionNN by minimizing cross-entropy loss against oracle decisions $y_{i}$. We optimize with AdaGrad Duchi et al. (2011). We learn input embeddings after initializing randomly. The hyperparameters of our network and training procedure are: the learning rate, the dimensionality of input embeddings, the weight decay parameter, the batch size, and the hidden state size of the LSTM. We tune via random search Bergstra and Bengio (2012), selecting parameters which achieve highest accuracy in predicting oracle decisions for the validation set. We train for 15 epochs, and we use parameters from the best-performing epoch (by validation accuracy) at test time.

A.3 Reimplementation of Filippova and Altun (2013)

In this work, we reimplement the method of Filippova and Altun (2013), who in turn implement a method partially described in Filippova and Strube (2008). There are inevitable discrepancies between our implementation and the methods described in these two prior papers.

Where the original authors train on only 100,000 sentences, we learn weights with the full training set to compare fairly with vertex addition (each model trains on the full training set). 2. 2.

We use Gurobi Optimization (2018) (v8) to solve the liner program. Filippova and Strube (2008) report using LPsolve.888http://sourceforge.net/projects/lpsolve 3. 3.

We implement with the common Universal Dependencies (UD, v1) framework Nivre et al. (2016). Prior work Filippova and Strube (2008) implements with older dependency formalisms Briscoe et al. (2006); de Marneffe et al. (2006). 4. 4.

In Table 1 of their original paper, Filippova and Altun (2013) provide an overview of the syntactic, structural, semantic and lexical features in their model. We implement every feature described in the table. We do not implement features which are not described in the paper. 5. 5.

Filippova and Altun (2013) augment edge labels in the dependency parse of $S$ as a preprocessing step. We reimplement this step using off-the-shelf augmented modifiers and augmented conjuncts available with the enhanced dependencies representation in CoreNLP Schuster and Manning (2016). 6. 6.

Filippova and Altun (2013) preprocess dependency parses by adding an edge between the root node and all verbs in a sentence.999This step ensures that subclauses can be removed from parse trees, and then merged together to create a compression from different clauses of a sentence. We found that replicating this transform literally (i.e. only adding edges from the original root to all tokens tagged as verbs) made it impossible for the ILP to recreate some gold compressions. (We suspect that this is due to differences in output from part-of-speech taggers). We thus add an edge between the root node and all tokens in a sentence during preprocessing, allowing the ILP to always return the gold compression.

We assess convergence of the ILP by examining validation F1 score on the traditional sentence compression task. We terminate training after six epochs, when F1 score stabilizes (i.e. changes by fewer than $10^{-3}$ points).

A.4 Implementation of SLOR

We use the SLOR function to measure the readability of the shortened sentences produced by each compression system. SLOR normalizes the probability of a token sequence assigned from a language model by adjusting for both the probability of the individual unigrams in the sentence and for the sentence length.101010Longer sentences are always less probable than shorter sentences; rarer words make a sequence less probable.

Following Lau et al. (2015), we define the function as

[TABLE]

where $\xi$ is a sequence of words, $P_{u}(\xi)$ is the unigram probability of this sequence of words and $P_{m}(\xi)$ is the probability of the sequence, assigned by a language model. $|\xi|$ is the length (in tokens) of the sentence.

We use a 3-gram language model trained on the training set of the Filippova and Altun (2013) corpus. We implement with KenLM Heafield (2011). Because compression often results in shortenings where the first token is not capitalized (e.g. a compression which begins with the third token in $S$) we ignore case when calculating language model probabilities.

A.5 Latency evaluation

To measure latency, for each technique, we sample 100,000 sentences with replacement from the test set. We observe the mean time to compress each sentence using Python’s built-in timeit module. In order to minimize effects from unanticipated confounds in measuring latency, we repeat this experiment three separate times (with a one hour delay between experiments). Thus in total we collect 300,000 observations for each compression technique. We observe that runtimes are log normal, and thus report each latency as the geometric mean of 300,000 observations. We use an Intel Xeon processor with a clock rate of 2.80GHz.

A.6 Compression ratios

When comparing sentence compression systems, it is important to ensure that all approaches use the same rate of compression Napoles et al. (2011). Following Filippova et al. (2015), we define the compression ratio as the character length of the compression divided by the character length of the sentence. We present test set compression ratios for all methods in Table 4. Because ratios are similar, our comparison is appropriate.