Efficient Representative Subset Selection over Sliding Windows

TL;DR

This paper introduces efficient algorithms for dynamic representative subset selection over sliding windows in data streams, supporting multiple constraints and demonstrating significant speedups with high-quality solutions.

Contribution

It formulates the data recency-aware RSS as a submodular maximization with multiple constraints and proposes the KW and KW+ frameworks for efficient, approximate solutions.

Findings

KW achieves over 100x speedup over batch methods.

KW+ is 5-10x faster than KW with comparable or better utility.

Both methods maintain high solution quality in real-world datasets.

Abstract

Representative subset selection (RSS) is an important tool for users to draw insights from massive datasets. Existing literature models RSS as the submodular maximization problem to capture the "diminishing returns" property of the representativeness of selected subsets, but often only has a single constraint (e.g., cardinality), which limits its applications in many real-world problems. To capture the data recency issue and support different types of constraints, we formulate dynamic RSS in data streams as maximizing submodular functions subject to general $d$-knapsack constraints (SMDK) over sliding windows. We propose a \textsc{KnapWindow} framework (KW) for SMDK. KW utilizes the \textsc{KnapStream} algorithm (KS) for SMDK in append-only streams as a subroutine. It maintains a sequence of checkpoints and KS instances over the sliding window. Theoretically, KW is $\frac{1-\varepsilon}{1+d}$-approximate for SMDK. Furthermore, we propose a \textsc{KnapWindowPlus} framework (KW$^{+}$) to improve upon KW. KW$^{+}$ builds an index \textsc{SubKnapChk} to manage the checkpoints and KS instances. \textsc{SubKnapChk} deletes a checkpoint whenever it can be approximated by its successors. By keeping much fewer checkpoints, KW$^{+}$ achieves higher efficiency than KW while still guaranteeing a $\frac{1-\varepsilon'}{2+2d}$-approximate solution for SMDK. Finally, we evaluate the efficiency and solution quality of KW and KW$^{+}$ in real-world datasets. The experimental results demonstrate that KW achieves more than two orders of magnitude speedups over the batch baseline and preserves high-quality solutions for SMDK over sliding windows. KW$^{+}$ further runs 5-10 times faster than KW while providing solutions with equivalent or even better utilities.

Tables

Table 1. Table 1: Frequently used notations

Notation	Description
$V,v_t$	$V$ is an unbounded stream of elements; $v_t\in V$ is the $t$-th element in the stream.
$d,\xi$	$d$ is the dimension of the knapsack constraint; $\xi$ is the family of sets defined by the $d$-knapsack constraint.
$c_j\left(v\right),c_{tj}$	$c_j\left(v\right)$ is the cost of $v$ in the $j$-th knapsack; $c_{tj}$ is the cost of $v_t$ in the $j$-th knapsack.
${\gamma }_t,{\delta }_t$	${\gamma }_t={\min }_{\forall j\in \left[d\right]}{c}_{tj}$ and ${\delta }_t={\max }_{\forall j\in \left[d\right]}{c}_{tj}$ are the minimum and maximum costs of $v_t$ in all $d$ knapsacks.
$\gamma ,\delta$	$\gamma ={\min }_{\forall t,\forall j}{c}_{tj}$ and $\delta ={\max }_{\forall t,\forall j}{c}_{tj}$ are the lower and upper bounds for the costs of any elements in the stream.
$f(\cdot ),{\Delta }_f(\cdot |\cdot )$	$f(\cdot )$ is a monotone submodular utility function; ${\Delta }_f(\cdot |\cdot )$ is the marginal gain defined on $f(\cdot )$.
$W$	$W$ is the size of the sliding window.
$A_t$	$A_t=⟨v_{t^{\prime }},\dots ,v_t⟩$ is the active window at time $t$ where $t^{\prime }=\max \{1,t-W+1\}$.
$S_t^{\ast },{\mathrm{𝙾𝙿𝚃}}_t$	$S_t^{\ast }$ is the optimal solution for SMDK w.r.t. the active window $A_t$ at time $t$; ${\mathrm{𝙾𝙿𝚃}}_t=f\left(S_t^{\ast }\right)$ denotes the optimal utility value.
$S_t$	$S_t$ denotes an approximate solution for SMDK w.r.t. $A_t$ at time $t$.
$X_t,x_i$	$X_t=⟨x_1,\dots ,x_s⟩$ is the sequence of $s$ checkpoints at time $t$ maintained by KW and ${\text{KW}}^{+}$; $x_i$ is the $i$-th checkpoint in $X_t$.
$S_{x,y}^{\ast },S_{x,y}$	$S_{x,y}^{\ast }$ and $S_{x,y}$ are the optimal solution and an approximate solution for SMDK w.r.t. a substream $V_{x,y}=⟨v_x,\dots ,v_y⟩$.

Table 2. Table 2: The parameters tested in the experiments

Parameter	Values
$d$	1, 2, 3, 4, 5
$c$	0.02, 0.04, 0.06, 0.08, 0.1
$W$	100k, 200k, 300k, 400k, 500k
$\lambda$	0.05, 0.1, 0.15, 0.2, 0.25
$\beta$	0.05, 0.1, 0.15, 0.2, 0.25

Table 3. Table 3: The number of checkpoints and elements (including candidates and buffers) maintained by KW + superscript KW \text{KW}^{+} .

Dataset	Parameter	$W$					$c$
	Value	100k	200k	300k	400k	500k	0.02	0.04	0.06	0.08	0.1
Twitter	#checkpoints	4.89	4.49	4.32	4.34	4.27	4.44	4.49	4.15	4.42	4.68
	#elements	3949.9	3770.4	3674.2	3725.3	3676.1	5489.4	3770.4	2843.4	2618.1	2509.5
Yahoo! Webscope	#checkpoints	4.44	3.58	3.62	3.5	2.8	4.4	3.58	3.18	3.44	2.7
	#elements	3258.6	2617.54	2735.38	2647.16	2019.9	6036.86	2617.54	1680.88	1454.92	908.16

Table 4. Table 4: A theoretical comparison of existing submodular maximization algorithms. The algorithms proposed in this work are highlighted by ∗ .

Algorithm	Data model	Constraint	Approximation	Time complexity
Sviridenko [31]	batch	$1$-knapsack	$1-\frac{1}{e}$	$𝒪\left(W^5\right)$
Kulik et al. [14]	batch	$d$-knapsack	$1-\frac{1}{e}-\epsilon$	$𝒪\left(W^{d\cdot {\epsilon }^{-4}}\right)$
Badanidiyuru et al. [1]	batch	$1$-knapsack	$1-\frac{1}{e}-\epsilon$	$𝒪\left(W^2\cdot {\left({\epsilon }^{-1}\cdot \log W\right)}^{{\epsilon }^{-8}}\right)$
Leskovec et al. [17] & Lin et al. [23]	batch	$1$-knapsack	$\frac{1}{2}\left(1-\frac{1}{e}\right)$	$𝒪\left({\gamma }^{-1}\cdot W\right)$
Badanidiyuru et al. [2] & Kumar et al. [15]	append-only stream	cardinality	$\frac{1}{2}-\epsilon$	$𝒪\left(\frac{\log k}{\epsilon }\right)$
Huang et al. [12]	append-only stream	$1$-knapsack	$\frac{4}{11}-\epsilon$	$𝒪\left({\left(\frac{\log {\gamma }^{-1}}{\epsilon }\right)}^4\right)$
Yu et al. [38]	append-only stream	$d$-knapsack	$\frac{1}{1+2d}-\epsilon$	$𝒪\left(\frac{\log {\gamma }^{-1}}{\epsilon }\right)$
Epasto et al. [6]	sliding window	cardinality	$\frac{1}{3}-\epsilon$	$𝒪\left(\frac{{\log }^2\left(k\cdot \theta \right)}{{\epsilon }^2}\right)$
KnapStream (KS)∗	append-only stream	$d$-knapsack	$\frac{1-\epsilon }{1+d}$	$𝒪\left(\frac{\log \left(d\cdot {\gamma }^{-1}\right)}{\epsilon }\right)$
KnapWindow (KW)∗	sliding window	$d$-knapsack	$\frac{1-\epsilon }{1+d}$	$𝒪\left(\frac{\sqrt{W}\cdot \log \left(d\cdot {\gamma }^{-1}\right)}{\epsilon }\right)$
KnapWindowPlus (${\text{KW}}^{+}$)∗	sliding window	$d$-knapsack	$\frac{1-{\epsilon }^{\prime }}{2+2d}$	$𝒪\left(\frac{\log \left(d\cdot {\gamma }^{-1}\right)}{{\epsilon }^{\prime }}\cdot \left({\gamma }^{-2}+\frac{\log \theta }{{\epsilon }^{\prime }}\right)\right)$

Equations

$$\underset{S_t\subseteq A_t}{\max }f(S_t)\text{s.t.}{S}_t\in \xi$$

$$f(S)=\sum _{w\in \mathcal{W}}\underset{v\in S}{\max }n(v,w)\cdot p(w)\cdot \log \frac{1}{p(w)}$$

$$f(S)=\frac{1}{2}\log \det (\mathbf{I}+{\sigma }^{-2}{\mathbf{K}}_{S,S})$$

$$f(S_{\varphi })=\sum _{i=1}^{|S_{\varphi }|}{\Delta }_f(s_i|S_{\varphi }^{i-1})\ge \frac{\varphi }{1+d}\cdot c_j(S_{\varphi })=\frac{\varphi }{1+d}\cdot b_j$$

$$f(S_{\varphi })\ge \frac{b\cdot \varphi }{1+d}\ge \frac{(1-\delta )\cdot \varphi }{1+d}\ge \frac{(1-\lambda )(1-\delta )}{1+d}\cdot \mathtt{OPT}$$

$$f(S_{\varphi }\cup S_j^{\ast })-f(S_{\varphi })\le \sum _{a\in S_j^{\ast }}{\Delta }_f(a|S_{\varphi })<\frac{\varphi \cdot c_j(S_j^{\ast })}{1+d}\le \frac{\varphi }{1+d}$$

$$f(S^{\ast }\cup S_{\varphi })-f(S_{\varphi })\le \sum _{j=1}^{d}f(S_{\varphi }\cup S_j^{\ast })-f(S_{\varphi })<\frac{d\varphi }{1+d}$$

$$f(S^{\prime}_{\phi}\cup\{a\})\geq\frac{\phi}{1+d}\cdot\Big{(}c_{j}(S^{\prime}_{\phi})+c_{j}(a)\Big{)}>\frac{\phi}{1+d}$$

$$\frac{\varphi }{1+d}\le f(S_{\varphi }^{\prime }\cup \{a\})\le f(S_{\varphi }^{\prime })+f(\{a\})\le f(S_{\varphi })+f(\{v_{max}\})$$

$$f[x_2,t]\ge \frac{(1-\beta )(1-\epsilon )}{1+d}\cdot {\mathtt{OPT}}_{t^{\prime }}^{x_1}\ge \frac{(1-\beta )(1-\epsilon )}{1+d}\cdot {\mathtt{OPT}}_1$$

$$f[x_2,t]\ge \frac{1-\epsilon }{1+d}\cdot {\mathtt{OPT}}_t^{x_2}\ge \frac{1-\epsilon }{1+d}\cdot {\mathtt{OPT}}_2$$

$$f(S_t)=f[x_2,t]\ge \frac{(1-\beta )(1-\epsilon )}{2(1+d)}\cdot {\mathtt{OPT}}_t$$

Full text

1 Introduction

In the big data era, a vast amount of data is being continuously generated by various applications, e.g., social media, network traffic, sensors, etc. An imperative task is to extract useful information from massive datasets. A compelling approach is representative subset selection [10, 23, 35, 37, 2, 36, 40, 24, 25, 26] (RSS): extracting a concise subset of representative elements from the source dataset. RSS is often formulated as selecting a subset of elements to maximize a utility function that quantifies the representativeness subject to some constraints. The utility functions are often chosen to be submodular to capture the “diminishing returns” property of representativeness [10, 2, 40, 32, 38, 7], i.e., adding more elements decreases the marginal representativeness. A number of constraints are used to restrict the selected subset in various ways. For example, a common approach to scaling kernel methods in nonparametric learning is active set selection [10, 2] that extracts a subset $S$ with the maximum information entropy as representatives. It restricts the size of $S$ to $k$ (a.k.a cardinality constraint) so that at most $k$ elements are selected for kernel training to reduce the computational costs while still retaining model quality. As another example, social data summarization [40, 32] selects a subset $S$ to best preserve the information in a collection of social posts. To restrict the summary size, two constraints are imposed: the number of selected posts in $S$, as well as their total length, is bounded. Additionally, the influence scores are also modeled as constraints so that more influential elements could be included in the summary [38].

In many cases, data is generated rapidly and only available as a stream [19, 11, 30, 18, 39]. To address the requirement for summarizing such datasets in real-time, RSS over data streams [29, 10, 2, 32] has been extensively studied in recent years. However, there are two major drawbacks that limit the deployment of existing approaches to many real-world applications. First, most of the streaming RSS algorithms only work with cardinality constraints, i.e., selecting a set of $k$ elements as representatives, and cannot support more complex constraints. As aforementioned, a number of RSS problems consider more general multi-knapsack (a.k.a. $d$-knapsack) constraints beyond cardinality [22, 23, 38, 26]. However, the algorithms that only support cardinality constraints cannot provide solutions with any quality assurances in more general cases. Second, existing methods are developed for the append-only setting where elements are only inserted into but never deleted from the stream and thus the freshness of solutions is ignored. Data streams are highly dynamic and keep evolving over time, where recent elements are more important than earlier ones. The sliding window [5] model that only considers the $W$ most recent elements is a natural way to capture such an essence. Although a number of RSS algorithms have been developed for append-only streams, RSS over sliding windows is still largely unexplored and, to the best of our knowledge, only one existing method [6] is proposed. It is not surprising that the method is also specific for cardinality constraints.

To address the limitations of existing methods, it requires general RSS frameworks that (i) support different types of submodular utility functions, (ii) work with more than one knapsack constraint, and (iii) extract a subset of representative elements over a sliding window efficiently.

In this paper, we formulate dynamic RSS in data streams as maximizing submodular functions with $d$-knapsack constraints (SMDK) over sliding windows. As SMDK is NP-hard, we focus on designing efficient approximation algorithms for SMDK. First, we devise the KnapStream algorithm (KS) for SMDK in append-only streams. KS needs a single pass over a stream and provides a $\frac{1-\varepsilon}{1+d}$-approximate solution for SMDK. It improves the state-of-the-art approximation factor of $\frac{1}{1+2d}-\varepsilon$ for SMDK in append-only streams [38]. Then, we propose two novel frameworks, namely KnapWindow (KW) and KnapWindowPlus ($\text{KW}^{+}$), for SMDK over sliding windows. Both frameworks adapt KS for the sliding window model by maintaining a sequence of KS instances starting at different timestamps (a.k.a checkpoints) over the sliding window. Specifically, KW maintains $\mathcal{O}(\sqrt{W})$ checkpoints over a size-$W$ sliding window. The interval between any neighboring checkpoints of KW is always equal. The approximation factor of KW for SMDK is the same as KS, i.e., $\frac{1-\varepsilon}{1+d}$. Furthermore, $\text{KW}^{+}$ is proposed to build an index SubKnapChk to manage the checkpoints based on their achieved utilities. SubKnapChk deletes a checkpoint whenever it can be approximated by its successors. Theoretically, the number of checkpoints in $\text{KW}^{+}$ is independent of $W$ and logarithmic to the range of the utility function. Since $\text{KW}^{+}$ maintains much fewer checkpoints, it achieves higher efficiency than KW. Nevertheless, $\text{KW}^{+}$ can still guarantee $\frac{1-\varepsilon^{\prime}}{2+2d}$-approximation solutions for SMDK over sliding windows.

Finally, we evaluate the efficiency and effectiveness of KW and $\text{KW}^{+}$ with two real-world applications: social stream summarization and active set selection. The experimental results show that KW achieves more than two orders of magnitude speedup over the batch baseline and preserves high-quality solutions for SMDK over sliding windows. $\text{KW}^{+}$ further runs 5–10 times faster than KW while providing solutions with equivalent or even better utilities.

Our main contributions are summarized as follows.

• •

We formulate dynamic RSS as maximizing submodular functions with $d$-knapsack constraints (aka SMDK) over sliding windows.

• •

We propose a novel $\frac{1-\varepsilon}{1+d}$-approximation KW framework for SMDK over sliding windows.

• •

We devise $\text{KW}^{+}$ to improve upon KW. Although the approximation factor of $\text{KW}^{+}$ drops to $\frac{1-\varepsilon^{\prime}}{2+2d}$, $\text{KW}^{+}$ has much higher efficiency than KW while providing solutions with equivalent or better quality.

• •

We demonstrate the efficiency and solution quality of KW and $\text{KW}^{+}$ for real-world applications.

The remaining of this paper is organized as follows. Section 2 defines dynamic RSS as SMDK over sliding windows. Section 3 gives two examples of modeling real-world RSS applications as SMDK. Section 4 and Section 5 present the KW and $\text{KW}^{+}$ frameworks respectively. Section 6 reports the experimental results. Section 7 reviews the related work. Finally, Section 8 concludes the whole paper.

2 Problem Formulation

In this section, we first introduce data streams and the sliding window model. Next, we give the notions of submodular functions and knapsack constraints. Then, we formally define the representative subset selection (RSS) problem as submodular maximization with a $d$-knapsack constraint (SMDK) in the sliding window model. Finally, we show the challenges of SMDK over sliding windows.

Data Stream & Sliding Window. A data stream comprises an unbounded sequence of elements $V=\langle v_{1},v_{2},\ldots\rangle$ and $v_{t}\in V$ is the $t$-th element of the stream. The elements in $V$ arrive one at a time in an arbitrary order. Only one pass over the stream is permitted and the elements must be processed in the arrival order. Specifically, we focus on the sliding window model for data streams. Let $W$ be the size of the sliding window. At any time $t$, the active window $A_{t}$ is a subsequence that always contains the $W$ most recent elements (a.k.a. active elements) in the stream111We only discuss the sequence-based sliding window in this paper. Nevertheless, the proposed algorithms can naturally support the time-based sliding window., i.e., $A_{t}=\langle v_{t^{\prime}},\ldots,v_{t}\rangle$ where $t^{\prime}=\max(1,t-W+1)$.

RSS over Sliding Windows. RSS selects a set of representative elements from the ground set according to a utility function with some budget constraint. In this paper, we target the class of nonnegative monotone submodular utility functions adopted in a wide range of RSS problems [2, 6, 10, 24, 32, 37, 40].

Given a ground set of elements $V$, we consider a set function $f:2^{V}\rightarrow\mathbb{R}_{\geq 0}$ that maps any subset of elements to a nonnegative utility value. For a set of elements $S\subseteq V$ and an element $v\in V\setminus S$, the marginal gain of $f(\cdot)$ is defined by $\Delta_{f}(v|S)\triangleq f(S\cup\{v\})-f(S)$. Then, the monotonicity and submodularity of $f(\cdot)$ can be defined according to its marginal gain.

Definition 1** (Monotonicity & Submodularity).**

A set function $f(\cdot)$ is monotone iff $\Delta_{f}(v|S)\geq 0$ for any $S\subseteq V$ and $v\in V\setminus S$. $f(\cdot)$ is submodular iff $\Delta_{f}(v|S)\geq\Delta_{f}(v|S^{\prime})$ for any $S\subseteq S^{\prime}\subseteq V$ and $v\in V\setminus S^{\prime}$.

Intuitively, monotonicity means adding more elements does not decrease the utility value. Submodularity captures the “diminishing returns” property that the marginal gain of adding any new element decreases as a set grows larger.

To handle various types of linear budget constraints in real-world problems, we adopt the general $d$-knapsack constraint [22, 23, 26, 38]. Specifically, a knapsack is defined by a cost function $c:V\rightarrow\mathbb{R}_{+}$ that assigns a positive cost to each element in the ground set $V$. Let $c(v)$ denote the cost of $v\in V$. The cost $c(S)$ of a set $S\subseteq V$ is the sum of the costs of its members, i.e., $c(S)=\sum_{v\in S}c(v)$. Given a budget $b$, we say $S$ satisfies the knapsack constraint iff $c(S)\leq b$. W.l.o.g., we normalize the budget to $b=1$ and the cost of any element to $c(v)\in(0,1]$. Then, a $d$-knapsack constraint $\xi$ is defined by $d$ cost functions $c_{1}(\cdot),\ldots,c_{d}(\cdot)$. Formally, we define $\xi=\{S\subseteq V:c_{j}(S)\leq 1,\forall j\in[d]\}$. We say a set $S$ satisfies the $d$-knapsack constraint iff $S\in\xi$.

Given the above definitions, we can formulate RSS as an optimization problem of maximizing a monotone submodular utility function $f(\cdot)$ subject to a $d$-knapsack constraint $\xi$ (SMDK) over the active window $A_{t}$. At every time $t$, RSS returns a subset of elements $S_{t}$ that (1) only contains active elements, (2) satisfies the $d$-knapsack constraint $\xi$, and (3) maximizes the utility function $f(\cdot)$. Formally,

[TABLE]

We use $S^{*}_{t}=\operatorname*{argmax}_{S_{t}\subseteq A_{t}:S_{t}\in\xi}f(S_{t})$ to denote the optimal solution of SMDK at time $t$.

Example 1**.**

A toy example of SMDK over sliding windows is given in Figure 1. We consider one of the simplest SMDK problems: budgeted maximum coverage (BMC) [13]. Given a domain of items $\mathcal{W}=\{w_{1},\ldots,w_{5}\}$, we have a sequence of sets $\mathcal{T}=\langle T_{1},\ldots,T_{5}\rangle$ where each set $T\in\mathcal{T}$ is a subset of $\mathcal{W}$ associated with two costs $c_{1}$ and $c_{2}$. Let the window size be $4$. The objective of BMC is to select a set of sets $S^{*}_{t}$ from $4$ most recent sets such that the number of items covered by $S^{*}_{t}$ is maximized while $S^{*}_{t}$ satisfies the $2$-knapsack constraint defined by $c_{1}$ and $c_{2}$. In Figure 1, we highlight two active windows $A_{4}$ and $A_{5}$ at time $4$ and $5$ respectively. Then, we give the optimal solutions $S^{*}_{4}$ and $S^{*}_{5}$ and their utilities for BMC at time $4$ and $5$.

Challenges of SMDK over Sliding Windows. SMDK is NP-hard. According to the definition of the $d$-knapsack constraint, the cardinality constraint with budget $k$ is a special case of a $1$-knapsack constraint when $c(v)=\frac{1}{k},\forall v\in V$. Because maximizing a submodular function with a cardinality constraint is NP-hard [28, 8], SMDK is NP-hard as well. Due to the submodularity of the utility function, a naïve approach to SMDK over sliding windows is storing the active window $A_{t}$ and rerunning a batch algorithm for SMDK on $A_{t}$ from scratch for every window slide. Typical batch algorithms for SMDK are CostEffectGreedy [23, 17] (CEG), an extension of the classic greedy algorithm [28], and ContinuousGreedy [14, 9] (ContG) which is based on the multi-linear relaxation technique. From the theoretical perspective, the approximation ratio of CEG for SMDK depends on the dimension of knapsacks $d$ while ContG can achieve a constant approximation (e.g., $1-\frac{1}{e}-\varepsilon$) independent of $d$. But ContG suffers from extremely high time complexity (e.g., $\mathcal{O}(W^{d\cdot\varepsilon^{-4}})$ [14], see Table 4) and is not practical even for very small $W$. In practice, we implement CEG as the batch baseline. CEG returns near-optimal solutions for SMDK empirically when the cost distribution is not extremely adversary [17, 38]. Nevertheless, for SMDK over sliding windows, CEG still needs to scan the active elements for multiple passes and incurs heavy computational costs. Hence, our challenge is to design efficient frameworks to continuously maintain the solutions for SMDK over sliding windows when new elements arrive rapidly, while guaranteeing a constant approximation ratio w.r.t. a fixed $d$.

Before moving on to the subsequent sections, we summarize the frequently used notations in Table 1.

3 Applications

In this section, we give two examples of RSS applications and describe how they are modeled as SMDK over sliding windows. The experiments for both applications in real-world datasets will be reported in Section 6. Note that many more RSS problems can also be modeled as SMDK (see Section 7), which could potentially benefit from this work.

3.1 Social Stream Summarization

Massive data is continuously generated as a stream by hundreds of millions of users on social platforms, e.g., Twitter. Social stream summarization aims to retain a small portion of representative elements from a user-generated stream. One common approach is topic-preserving summarization [40, 32] that selects a subset of posts that best preserve latent topics in the stream. We focus on topic-preserving summarization in the sliding window model to capture the evolving nature of social streams, i.e., topics under discussion change over time [32]. We consider a collection of social posts $V$ is available as a stream in ascending order of timestamp. A social post $v\in V$ is represented as a bag of $l$ words $\{w_{1},\ldots,w_{l}\}$ drawing from the vocabulary $\mathcal{W}$. The utility $f(S)$ for a set of elements $S$ is computed by summing up the weights of words in $S$ where the weight of a word $w$ is acquired based on its information entropy [40]. Specifically,

[TABLE]

where $n(v,w)$ is the frequency of word $w$ in element $v$, $p(w)=\frac{\sum_{v\in V}n(v,w)}{\sum_{v\in V}\sum_{w\in\mathcal{W}}n(v,w)}$ is the probability of generating a word $w$ from the topic model. $f(S)$ has been proved to be monotone and submodular [40]. Furthermore, the representatives should satisfy the following $3$-knapsack constraint. First, a uniform cost $c_{1}(v)$ is assigned to each element $v\in V$, i.e., $c_{1}(v)=\frac{1}{k}$, to bound the size of the representative set within $k$ [40, 32]. Second, a cost $c_{2}(v)$ is assigned to the length $l$ of element $v$ since users prefer shorter summaries to longer ones [23, 22]. For normalization, we compute the average number of words $\overline{l}$ in one element and assign $c_{2}(v)$ as follows: given an element $v$ of $l$ words, $c_{2}(v)=\frac{1}{k}\cdot\frac{l}{\overline{l}}$. For example, when $\overline{l}=5,k=10$, an element $v$ with $l=10$ words has a cost $c_{2}(v)=0.2$. Third, a cost $c_{3}(v)$ is assigned according to social influence [40, 38]. Let $fl(v)$ denote the number of followers of the user who posts $v$. We consider $c_{3}(v)=\min(\delta,\frac{1}{k}\cdot\frac{\log(1+fl(v))}{\log(1+\overline{fl})})$ where $\overline{fl}$ is the average number of followers of each user and $\delta$ is the upper-bound cost. We assign lower costs to the elements posted by more influential users so that the summary could include more influential elements. The upper-bound cost $\delta$ is assigned to elements posted by users with very few (e.g., 0 or 1) followers for normalization. To sum up, the social stream summarization is modeled as maximizing $f(\cdot)$ in Equation 2 with a $3$-knapsack constraint defined by $c_{1}(\cdot)$, $c_{2}(\cdot)$, and $c_{3}(\cdot)$ over the active window $A_{t}$.

3.2 Active Set Selection

Active set selection [2, 10] is a common approach to scaling kernel methods to massive datasets. It aims to select a small subset of elements with the maximal information entropy from the source dataset. In some sites like Yahoo!, weblogs are continuously generated by users as a stream. Given a stream of weblogs $V$, each record $v\in V$ is modeled as a multi-dimensional feature vector. The representativeness of a set of vectors $S$ is measured by the Informative Vector Machine [16] (IVM):

[TABLE]

where $\mathbf{K}_{S,S}$ is an $|S|\times|S|$ kernel matrix indexed by $S$ and $\sigma>0$ is a regularization parameter. For each pair of elements $v_{i},v_{j}\in S$, the $(i,j)$-th entry $\mathcal{K}_{i,j}$ of $\mathbf{K}$ represents the similarity between $v_{i}$ and $v_{j}$ measured via a symmetric positive definite kernel function. We adopt the squared exponential kernel embedded in the Euclidean space, i.e., $\mathcal{K}_{i,j}=\exp(-\frac{\lVert v_{i}-v_{j}\rVert^{2}_{2}}{h^{2}})$. It has been proved that $f(\cdot)$ in Equation 3 is a monotone submodular function [10]. Furthermore, other than assigning a fixed cost to each feature vector, existing methods also use different schemes to assign costs, e.g., generating from a Gamma distribution or marginal-dependent costs [4]. Thus, we consider a more general case: each feature vector $v$ is associated with a cost $c(v)$ drawing from an arbitrary distribution $\mathcal{D}$ within range $(0,1)$. The objective is to select a subset $S$ of feature vectors such that $f(S)$ in Equation 3 is maximized subject to a $1$-knapsack constraint defined by $c(\cdot)$ over the active window $A_{t}$.

4 The KnapWindow Framework

In this section, we propose the KnapWindow (KW) framework for SMDK over sliding windows. The architecture of KW is illustrated in Figure 2. KW always stores all active elements in $A_{t}$ at any time $t$. Then, KW adapts the KnapStream (KS) algorithm that provides an approximation solution for SMDK in append-only streams to work in the sliding model in the following manner. It maintains a sequence of checkpoints $X_{t}=\langle x_{1},\ldots,x_{s}\rangle\subseteq[t^{\prime},t]$ over the active window $A_{t}$. The interval between any neighboring checkpoints $x_{i}$ and $x_{i+1}$ is equal (e.g., the interval is $3$ in Figure 2). For each checkpoint $x_{i}$, a KS instance $\mathcal{H}(x_{i})$ is maintained by processing an append-only stream from $v_{x_{i}}$ to $v_{t}$. To retrieve the solution for SMDK at time $t$, KW always uses the result from $\mathcal{H}(x_{1})$ corresponding to $x_{1}$. $\mathcal{H}(x_{1})$ first post-processes the active elements before $v_{x_{1}}$ (e.g., the solid red ones in Figure 2) and uses the result after post-processing as the final solution.

The scheme of KS to maintain a solution for SMDK over an append-only stream is also illustrated in Figure 2. First, KS approximates the optimal utility $\mathtt{OPT}$ for SMDK by a sequence of estimations. Then, KS maintains a candidate for each estimation with a unique threshold derived from the estimation. Whenever receiving a new element, KS checks whether it can be included into each candidate independently according to the threshold. Finally, KS selects the candidate with the maximum utility among all candidates as the solution for its processed substream.

Next, Section 4.1 will present the KnapStream algorithm for SMDK in append-only streams. Then, Section 4.2 will introduce how the KnapWindow algorithm adapts KnapStream for the sliding window model. Finally, Section 4.3 will analyze both algorithms theoretically.

4.1 The KnapStream Algorithm

In this subsection, we propose the KnapStream (KS) algorithm to maintain a solution for SMDK w.r.t. an append-only stream $V_{x,y}=\langle v_{x},\ldots,v_{y}\rangle$ from time $x$ to $y$. KS follows the threshold-based framework [2, 15] for streaming submodular maximization. Its mechanism depends on estimating the optimal utility value $\mathtt{OPT}$ for SMDK w.r.t. $V_{x,y}$. Although $\mathtt{OPT}$ cannot be exactly determined unless P=NP, KS tracks the lower and upper bounds for $\mathtt{OPT}$ from the observed elements online and maintains a sequence of candidates with different estimations for $\mathtt{OPT}$ in the range. Each candidate derives a unique threshold for the marginal gain according to its estimation for $\mathtt{OPT}$. When a new element arrives, a candidate decides whether to include it based on the marginal gain of adding it into the candidate and the candidate’s threshold. After processing the stream, the candidate with the maximum utility is used as the solution.

Although having a similar scheme, the algorithms in [2] and [15] only work with one cardinality constraint, whereas KS is different from them in two aspects to achieve an approximation guarantee for general $d$-knapsack constraints: (1) the criterion for the inclusion of an element considers not only its marginal gain but also its costs, i.e., it checks the cost-effectiveness of adding the element in each knapsack and includes it only when its cost-effectiveness reaches the threshold in $d$ knapsacks; (2) the singleton element with the maximum self-utility is also a candidate solution.

The pseudo-code of KS is presented in Algorithm 1. Three auxiliary variables are maintained by KS (Lines 7–10): $v_{max}$ stores the element with the maximum self-utility; $M$ and $m$ track the upper and lower bounds for $\mathtt{OPT}$. Specifically, $M$ is the maximum cost-effectiveness any observed element can achieve and $m$ is the corresponding self-utility. We will explain why they are the upper and lower bounds for $\mathtt{OPT}$ in the proof of Theorem 1. The sequence of estimations $\Phi=\langle(1+\lambda)^{l}|l\in\mathbb{Z},m\leq(1+\lambda)^{l}\leq M\cdot(1+d)\rangle$ and corresponding candidates are updated based on the up-to-date $m$ and $M$ (Lines 11–12). Then, given an element $v_{t}$, each candidate checks whether to include it independently. For each $\phi\in\Phi_{t}$, if the marginal gain $\Delta_{f}(v_{t}|S_{\phi})$ of adding $v_{t}$ to $S_{\phi}$ reaches $\frac{\delta_{t}\cdot\phi}{1+d}$ where $\delta_{t}=\max_{\forall j\in[d]}c_{tj}$ and the $d$-knapsack constraint is still satisfied after adding $v_{t}$, $v_{t}$ will be included into $S_{\phi}$ (Lines 13–15). Finally, after processing every element in the stream, it first finds $S_{max}$ with the maximum utility among the candidates and then compares the utility of $S_{max}$ with that of $\{v_{max}\}$. The one with the higher utility is returned as the solution $S_{x,y}$ for SMDK w.r.t. the stream $V_{x,y}$ (Lines 16 and 17).

4.2 The KnapWindow Algorithm

In this subsection, we present the KnapWindow (KW) algorithm. It adapts KS for SMDK in the sliding window model by maintaining a sequence of checkpoints and corresponding KS instances over the sliding window. At any time $t$, KW maintains a sequence of $s$ checkpoints $X_{t}=\langle x_{1},\ldots,x_{s}\rangle\subseteq[t^{\prime},t]$. The interval between any neighboring checkpoints in $X_{t}$ is always equal. Given the interval $L\in\mathbb{Z}^{+}$, KW only creates a new checkpoint and initiates a new KS instance for every $L$ elements. For each checkpoint $x_{i}$, a KS instance $\mathcal{H}(x_{i})$ is maintained by processing a substream from element $v_{x_{i}}$ to the up-to-date element $v_{t}$. Whenever the first checkpoint $x_{1}$ expires from the sliding window ($x_{1}<t^{\prime}$ where $t^{\prime}=max(1,t-W+1)$), it will be deleted from $X_{t}$. The corresponding KS instance $\mathcal{H}(x_{1})$ will be terminated as well. To provide the solution for SMDK w.r.t. $A_{t}$, it uses the result from $\mathcal{H}(x_{1})$. But it is noted that the elements from $v_{t^{\prime}}$ to $v_{x_{1}-1}$ have not been processed by $\mathcal{H}(x_{1})$ yet. Therefore, it feeds the unprocessed elements to $\mathcal{H}(x_{1})$ before returning the final solution.

The pseudo-code of KW is presented in Algorithm 2. The sequence of checkpoints is initialized to $X_{0}=\varnothing$. A checkpoint $x_{s}=t$ is created and appended to the end of $X_{t}$ at time $t=L,2L,\ldots$. A KS instance $\mathcal{H}(x_{s})$ is initiated accordingly (Lines 5–7). Then, it deletes the expired checkpoints from $X_{t}$ (Lines 8–10). Subsequently, each checkpoint processes $v_{t}$ and updates the result independently. This procedure follows Lines 8–15 of Algorithm 1. To provide the solution $S_{t}$ for SMDK at time $t$, $\mathcal{H}(x_{1})$ post-processes the elements from $v_{t^{\prime}}$ to $v_{x_{1}-1}$ (Line 14). Finally, the solution of $\mathcal{H}(x_{1})$ after post-processing is returned as $S_{t}$ (Line 15).

4.3 Theoretical Analysis

In this subsection, we analyze the approximation ratios and complexities of KS and KW. In the theoretical analysis, we assume the cost of any element is bounded by $\gamma$ and $\delta$, i.e., $0<\gamma\leq c_{tj}\leq\delta\leq 1$ for all $t,j$. It is noted that the algorithms do not need to know $\gamma$ and $\delta$ in advance.

The roadmap of our analysis is as follows. First of all, we present the approximation ratio of KS. We first show that if we knew the optimal utility $\mathtt{OPT}$ for SMDK w.r.t. $V_{x,y}$ in advance, the candidate whose estimation is the closest to $\mathtt{OPT}$ would be a $\frac{(1-\lambda)(1-\delta)}{1+d}$ approximate solution (Lemma 1). However, the approximation ratio depends on $\delta$ and may degrade arbitrarily when $\delta$ increases. Therefore, we further show that if the singleton element with the maximum self-utility is also considered as a candidate solution (Line 16 of Algorithm 1), there is a lower bound for the approximation ratio regardless of $\delta$ (Lemma 2). Then, as $\mathtt{OPT}$ is unknown unless P=NP, we analyze how KS can track the lower and upper bounds for $\mathtt{OPT}$ and how many different estimations are required to guarantee that at least one of them approximates $\mathtt{OPT}$ within a bounded error ratio (Theorem 1). As KS maintains one candidate for each $\mathtt{OPT}$ estimation, we can get its time and space complexity accordingly. After providing the theoretical results for KS, we extend these results to KW. Specifically, KW retains the approximation ratio of KS because it is guaranteed that the solution of KW is returned only after processing all active elements (Theorem 2). Finally, we analyze the complexity of KW.

Lemma 1**.**

Assuming there exists $\phi\in\Phi$ such that $(1-\lambda)\mathtt{OPT}\leq\phi\leq\mathtt{OPT}$ where $\mathtt{OPT}$ is the optimal utility of SMDK w.r.t. $V_{x,y}$, $S_{\phi}$ satisfies that $f(S_{\phi})\geq\frac{(1-\lambda)(1-\delta)}{1+d}\cdot\mathtt{OPT}$.

Proof.

Let $s_{i}$ be the $i$-th element added to $S_{\phi}$, $S^{i}_{\phi}$ be $\{s_{1},\ldots,s_{i}\}$ for $i\in[0,|S_{\phi}|]$ with $S^{0}_{\phi}=\emptyset$, $b_{j}=c_{j}(S_{\phi})$ for $j\in[d]$ be the cost of $S_{\phi}$ in the $j$-th knapsack, and $b=\max_{j\in[d]}b_{j}$ be the maximal cost of $S_{\phi}$ among $d$ knapsacks. According to Line 14 in Algorithm 1, we have $\Delta_{f}(s_{i}|S^{i-1}_{\phi})\geq\frac{c_{j}(s_{i})\cdot\phi}{1+d}$ for $j\in[d]$. It holds that:

[TABLE]

Therefore, $f(S_{\phi})\geq\frac{\phi}{1+d}\cdot b$.

Next, we discuss two cases separately as follows.

Case 1. When $b\geq(1-\delta)$, we have:

[TABLE]

Case 2. When $b<(1-\delta)$, we have $\forall v\in V\setminus S_{\phi}$, $S_{\phi}\cup\{v\}\in\xi$. Let $S^{*}$ be the optimal solution for $V$ and $a$ be an element in $S^{*}\setminus S_{\phi}$. Since $a$ is not added to $S_{\phi}$, there must exist $\mu(a)\in[d]$ such that $\Delta_{f}(a|S^{\prime}_{\phi})<\frac{c_{\mu(a)}(a)\cdot\phi}{1+d}$, where $S^{\prime}_{\phi}\subseteq S_{\phi}$ is the subset of $S_{\phi}$ when $a$ is processed. We consider $S^{*}_{j}=\{a|a\in S^{*}\setminus S_{\phi}\wedge\mu(a)=j\}$ for $j\in[d]$. Due to the submodularity of $f(\cdot)$, we acquire:

[TABLE]

Then, because $S^{*}\setminus S_{\phi}=\cup^{d}_{j=1}S^{*}_{j}$, we have:

[TABLE]

Finally, we get $f(S_{\phi})>\mathtt{OPT}-\frac{d}{1+d}\mathtt{OPT}\geq\frac{1}{1+d}\cdot\mathtt{OPT}$.

Considering both cases, we conclude the proof. ∎

Lemma 1 has proved that KS achieves a good approximation ratio when $\delta$ is small. Next, we further analyze the case where $\delta>0.5$ and prove that the approximation ratio has a lower bound regardless of $\delta$.

Lemma 2**.**

When $\delta>0.5$, it satisfies that at least one of $f(S_{\phi})$ and $f(\{v_{max}\})$ is greater than $\frac{0.5(1-\lambda)}{1+d}\cdot\mathtt{OPT}$.

Proof.

Lemma 2 naturally follows when $b\geq 0.5$ (Case 1 of Lemma 1) or for all $a\in S^{*}\setminus S_{\phi}$, $a$ is excluded from $S_{\phi}$ because its marginal gain does not reach the threshold in some knapsack (Case 2 of Lemma 1).

Thus, we only need to consider the following case: there exists some elements whose marginal gains reach the threshold in all knapsacks but are excluded from $S_{\phi}$ because including them into $S_{\phi}$ violates the $d$-knapsack constraint. Assuming $a$ is such an element for $S_{\phi}$, we have $\Delta_{f}(a|S^{\prime}_{\phi})\geq\frac{c_{j}(a)\cdot\phi}{1+d}$ and $c_{j}(S^{\prime}_{\phi})+c_{j}(a)>1$ for some $j\in[d]$. In this case, we have:

[TABLE]

Due to the monotonicity and submodularity of $f(\cdot)$, we get:

[TABLE]

Therefore, at least one of $f(S_{\phi})$ and $f(\{v_{max}\})$ is greater than $\frac{0.5\phi}{1+d}\cdot\mathtt{OPT}$ and we conclude the proof. ∎

Given Lemmas 1 and 2, we prove that KS achieves an approximation factor of $\frac{(1-\delta)(1-\lambda)}{1+d}$ (when $\delta\leq 0.5$) or $\frac{0.5(1-\lambda)}{1+d}$ (when $\delta>0.5$).

Theorem 1**.**

The solution $S_{x,y}$ returned by Algorithm 1 satisfies $f(S_{x,y})\geq\frac{1-\varepsilon}{1+d}\cdot f(S^{*}_{x,y})$ where $S^{*}_{x,y}$ is the optimal solution for SMDK w.r.t. $V_{x,y}$ and $\varepsilon=\min(\delta+\lambda,0.5+\lambda)$.

Proof.

By Lemmas 1 and 2, we can say Theorem 1 naturally holds if there exists at least one $\phi\in\Phi$ such that $(1-\lambda)\mathtt{OPT}\leq\phi\leq\mathtt{OPT}$. First, we show $m$ and $M$ are the lower and upper bounds for $\mathtt{OPT}$. It is easy to see $m\leq\mathtt{OPT}$ as $m\leq f(\{v_{max}\})$ and $\{v\}\in\xi$ for any $v\in V$. $M$ maintains the maximum cost-effectiveness among all elements. We have $M\geq\frac{f(\{v_{i}\})}{c_{ij}}$, $\forall i\in[x,y]$ and $\forall j\in[d]$. Let $S^{*}_{x,y}=\{a_{1},\ldots,a_{|S^{*}|}\}$ be the optimal solution for $V_{x,y}$. As $f(\cdot)$ is monotone submodular, $\mathtt{OPT}\leq\sum^{|S^{*}_{x,y}|}_{i=1}f(\{a_{i}\})\leq c_{j}(S^{*})M$ for $j\in[d]$. As $c_{j}(S^{*}_{x,y})\leq 1$, we have $M\geq\mathtt{OPT}$. KS estimates $\mathtt{OPT}$ by a sequence $\langle(1+\lambda)^{l}|l\in\mathbb{Z},m\leq(1+\lambda)^{l}\leq M(1+d)\rangle$. Then, there exists at least one estimation $\phi$ such that $\phi\leq\mathtt{OPT}\leq(1+\lambda)\phi$. Equivalently, $(1-\lambda)\mathtt{OPT}\leq\phi\leq\mathtt{OPT}$. Therefore, we conclude the proof by combining this result with Lemma 1 and 2. ∎

The Complexity of KS. As only one pass over the stream is permitted, to avoid missing elements with marginal gains of greater than $\frac{M}{1+d}$, KS maintains the candidates for estimations within an increased range $[m,(1+d)M]$ instead of $[m,M]$. Then, because $\frac{m}{M}\leq\gamma$ (Line 10 of Algorithm 1), the number of candidates in KS is bounded by $\lceil\log_{1+\lambda}\gamma^{-1}(1+d)\rceil$. Thus, we have KS maintains $\mathcal{O}(\frac{\log(d\cdot\gamma^{-1})}{\varepsilon})$ candidates. For each candidate, one function call is required to evaluate whether to add a new element. Thus, the time complexity to update one element is $\mathcal{O}(\frac{\log(d\cdot\gamma^{-1})}{\varepsilon})$. Finally, at most $\gamma^{-1}$ elements can be maintained in each candidate. Otherwise, the $d$-knapsack constraint must not be satisfied. Therefore, the number of elements stored is $\mathcal{O}(\frac{\log(d\cdot\gamma^{-1})}{\gamma\cdot\varepsilon})$.

Next, we present the approximation factor of KW.

Theorem 2**.**

The solution $S_{t}$ returned by Algorithm 2 satisfies $f(S_{t})\geq\frac{1-\varepsilon}{1+d}\cdot\mathtt{OPT}_{t}$ where $\mathtt{OPT}_{t}$ is the optimal utility for SMDK w.r.t. $A_{t}$ at time $t$ and $\varepsilon=\min(\delta+\lambda,0.5+\lambda)$.

It is obvious that $\mathcal{H}(x_{1})$ must have processed every element in $A_{t}$ after post-processing. As the approximation ratio of KS is order-independent, i.e., no assumption is made for the arrival order of elements, Theorem 2 holds.

The Complexity of KW. KW maintains $s=\lceil\frac{W}{L}\rceil$ checkpoints for $A_{t}$ and thus updates the KS instances for an element in $\mathcal{O}(\frac{s\cdot\log(d\cdot\gamma^{-1})}{\lambda})$ time. In addition, it takes $\mathcal{O}(\frac{L\cdot\log(d\cdot\gamma^{-1})}{\lambda})$ time for post-processing. The time complexity of KW is $\mathcal{O}(\frac{(s+L)\cdot\log(d\cdot\gamma^{-1})}{\lambda})$. When $s=L=\sqrt{W}$, it becomes $\mathcal{O}(\frac{\sqrt{W}\cdot\log(d\cdot\gamma^{-1})}{\lambda})$. Finally, because all active elements must be stored, the space complexity of KW is $\mathcal{O}(W)$.

5 The KnapWindowPlus Framework

Although KW can provide approximation solutions for SMDK with a theoretical bound, it still suffers from two drawbacks that limit its application for a large window size $W$. First, KW cannot handle the case when the window does not fit in the main memory. Second, as $\mathcal{O}(\sqrt{W})$ KS instances are maintained for a size-$W$ sliding window, the efficiency of KW degrades with increasing $W$. To improve upon KW, we further propose the KnapWindowPlus framework ($\text{KW}^{+}$) in this section.

The architecture of $\text{KW}^{+}$ is illustrated in Figure 3. The basic idea of $\text{KW}^{+}$ is similar to KW: it also keeps the sequence of checkpoints $X_{t}=\langle x_{1},\ldots,x_{s}\rangle$ and maintains a KS instance to process a substream from $v_{x_{i}}$ to $v_{t}$ at time $t$ in each checkpoint $x_{i}\in X_{t}$. However, $\text{KW}^{+}$ is substantially different from KW in the following four aspects. First, $\text{KW}^{+}$ does not store the entire active window but only keeps the elements within each KS instance. The number of elements kept by $\text{KW}^{+}$ is empirically much smaller than $W$. Second, $\text{KW}^{+}$ builds an index SubKnapChk for checkpoint maintenance. Instead of maintaining a sequence of checkpoints with equal interval, $\text{KW}^{+}$ creates a checkpoint and the corresponding KS instance for every arrival element. Then, SubKnapChk manages the checkpoints based on their utilities and deletes a checkpoint whenever it can be approximated by its successors. By using SubKnapChk, the number of checkpoints in $\text{KW}^{+}$ is independent of $W$. Third, $\text{KW}^{+}$ will keep one expired checkpoint (i.e., $x_{1}$) when $t>W$. It tracks the optimal utility $\mathtt{OPT}_{t}$ for SMDK w.r.t. $A_{t}$ to guarantee the theoretical soundness of the solutions. Fourth, $\text{KW}^{+}$ maintains a buffer with tunable size along with each candidate of the KS instances. In the post-processing procedure, the elements in buffers are added into the candidates to improve the utilities of solutions.

Next, Section 5.1 will introduce the $\text{KW}^{+}$ algorithm. Then, Section 5.2 will provide a theoretical analysis for $\text{KW}^{+}$. Finally, Section 5.3 will discuss how to adapt KW and $\text{KW}^{+}$ to the scenario where the sliding window shifts for more than one element at a time.

5.1 The KnapWindowPlus Algorithm

In this subsection, we describe the KnapWindowPlus algorithm ($\text{KW}^{+}$) in detail. We first present a novel index Submodular Knapsack Checkpoints (SubKnapChk) to maintain a sequence of checkpoints and corresponding KS instances over the sliding window. Then, we show the procedures for buffer maintenance and post-processing.

Submodular Knapsack Checkpoints. At time $t$, an index called Submodular Knapsack Checkpoints (SubKnapChk) comprises a sequence of $s$ checkpoints $X_{t}=\langle x_{1},\ldots,x_{s}\rangle$ where $x_{1}<\ldots<x_{s}=t$. For each checkpoint $x_{i}$, a KS instance $\mathcal{H}(x_{i})$ is maintained. $\mathcal{H}(x_{i})$ processes a substream from $v_{x_{i}}$ to $v_{t}$ and will be terminated when $x_{i}$ is deleted from SubKnapChk. When $t>W$, the first checkpoint $x_{1}$ expires (i.e., $x_{1}<t^{\prime}$) but is not deleted from SubKnapChk immediately. It is maintained to track the upper bound for the optimal utility $\mathtt{OPT}_{t}$ of SMDK w.r.t. $A_{t}$. However, the result of $\mathcal{H}(x_{1})$ cannot be used as the solution at time $t$ in this case because it may contain expired elements. SubKnapChk restricts the number of expired checkpoints to at most $1$. Therefore, $x_{2}$ must not expire and the result of $\mathcal{H}(x_{2})$ is returned as the solution for SMDK w.r.t. $A_{t}$ when $t>W$.

The idea of maintaining a sequence of checkpoints over sliding windows is inspired by smooth histograms [3]. However, according to the analysis in [6], the method in [3] cannot be directly applied to SMDK because it requires an append-only streaming algorithm with at least $0.8$-approximation for each checkpoint. Unfortunately, [28, 8] show that there is no polynomial algorithm for SMDK that can achieve an approximation ratio of better than $1-\frac{1}{e}\approx 0.63$ unless P=NP. Therefore, we devise a novel strategy to maintain an adequate sequence of checkpoints so that (1) the number of checkpoints is as few as possible for high efficiency; (2) the utilities of the solutions still achieve a bounded approximation ratio to the optimal one.

Towards both objectives, we propose the following strategy to maintain the checkpoints in SubKnapChk: (1) create a checkpoint and a KS instance for each arrival element; (2) delete a checkpoint and terminate its KS instance once it can be approximated by any successive checkpoint. Let $f[x_{i},t]$ denote the utility of the solution returned by $\mathcal{H}(x_{i})$ at time $t$. Given three neighboring checkpoints $x_{i},x_{i+1},x_{i+2}$ ($i\in[1,s-2]$) and a parameter $\beta>0$, if $f[x_{i+2},t]\geq(1-\beta)f[x_{i},t]$, we consider the second checkpoint $x_{i+1}$ can be approximated by the third one $x_{i+2}$. In this case, $x_{i+1}$ will be deleted from SubKnapChk. We will formally analyze the soundness of such a strategy in Section 5.2.

Buffer Maintenance and Post-Processing. To further improve the empirical performance of $\text{KW}^{+}$, we maintain buffers along with the candidates in KS instances and use these buffers for post-processing before returning the final solution. The reasons why the buffers and post-processing are essential are as follows. First, by using SubKnapChk, the solutions of $\mathcal{H}(x_{2})$ are always used for $A_{t}$ when $t>W$. As it is common that $x_{2}\gg t^{\prime}$, all elements between $v_{t^{\prime}}$ and $v_{x_{2}-1}$ are missing from the solutions of $\mathcal{H}(x_{2})$. Second, the candidates with high thresholds in KS instances are hard to be filled, even if more elements could still be added without violating the $d$-knapsack constraint. Therefore, we maintain the buffers for post-processing to improve the solution quality of $\text{KW}^{+}$.

We consider a buffer $B_{\phi}=\emptyset$ is initialized when each candidate $S_{\phi}$ in a KS instance $\mathcal{H}(x_{i})$ ($i\in[1,s]$) is created. When processing an element $v_{t}$, if adding $v_{t}$ to $S_{\phi}$ achieves a marginal gain of slightly lower than the threshold, i.e., $\Delta_{f}(v_{t}|S_{\phi})\geq\alpha\cdot\frac{\delta_{t}\cdot\phi}{1+d}$, $v_{t}$ will be added to $B_{\phi}$. Here, $\alpha\in(0,1)$ is used to control the lower bound for an element to be added to $B_{\phi}$. Furthermore, we restrict the buffer size to $\eta$. When the number of elements in $B_{\phi}$ exceeds $\eta$, we first drop each element $v$ if $S_{\phi}\cup\{v\}\notin\xi$. Then, we drop the elements with the least cost-effectivenesses w.r.t. $S_{\phi}$ until $|B_{\phi}|=\eta$, where the cost-effectiveness of element $v$ is computed by $\frac{\Delta_{f}(v|S_{\phi})}{\delta(v)}$, $\delta(v)=\max_{j\in[d]}c_{j}(v)$. Before returning the solution at time $t$, we perform the post-processing procedure using buffers of $\mathcal{H}(x_{1})$ and $\mathcal{H}(x_{2})$ (if $t<W$, only the buffers of $\mathcal{H}(x_{1})$ is used). Specifically, for each candidate $S_{\phi}$, we run CostEffectGreedy [23] to add elements in buffers to $S_{\phi}$. After post-processing each candidate, we also return the candidate with the maximum utility as the final solution.

Algorithmic Description. The pseudo-code of $\text{KW}^{+}$ is presented in Algorithm 3. The maintenance of SubKnapChk is shown in Lines 5–17. At time $t$, a new checkpoint $x_{s}=t$ and a KS instance $\mathcal{H}(x_{s})$ are created for $v_{t}$. Then, if there is more than one expired checkpoint in SubKnapChk, all except the last one will be deleted (Lines 8–10). This guarantees that there is only one expired checkpoint in $X_{t}$. Subsequently, for each checkpoint $x_{i}$, $\mathcal{H}(x_{i})$ processes $v_{t}$ and updates the candidates independently according to Lines 8–15 of Algorithm 1. After updating the candidates of $\mathcal{H}(x_{i})$ for $v_{t}$, it performs the buffer maintenance procedure as follows (Lines 27–33). If $\Delta_{f}(v_{t}|S_{\phi})\geq\alpha\cdot\frac{\delta_{t}\cdot\phi}{1+d}$, $v_{t}$ is added to $B_{\phi}$. When the number of elements in $B_{\phi}$ exceeds $\eta$, it first drops any $v\in B_{\phi}$ if $S_{\phi}\cup\{v\}\notin\xi$ and then drops the element $v^{\prime}$ with the least cost-effectiveness in $B_{\phi}$ until $|B_{\phi}|=\eta$. Next, it maintains the checkpoints in SubKnapChk. The checkpoints that can be approximated by its successor are identified and deleted from SubKnapChk (Lines 15–17). After the SubKnapChk maintenance, for any $x\in X_{t}$, there is at most one checkpoint $x^{\prime}\in X_{t}$ such that $x^{\prime}>x$ and $f[x^{\prime},t]\geq(1-\beta)f[x,t]$. Finally, the post-processing procedure is executed before returning the solution $S_{t}$ for SMDK w.r.t. $A_{t}$. When $t<W$, $\mathcal{H}(x_{1})$ will provide $S_{t}$. Each candidate $S_{\phi}$ in $\mathcal{H}(x_{1})$ considers $B_{\phi}$ for post-processing. Otherwise, $\mathcal{H}(x_{2})$ will provide $S_{t}$. We first add the non-expired elements in $S^{\prime}_{\phi}$ and $B^{\prime}_{\phi}$ of $\mathcal{H}(x_{1})$ to $B_{\phi}$ for post-processing. Starting from $S_{\phi}$, the post-processing procedure greedily adds the element $v^{*}$ with the maximum cost-effectiveness in $B_{\phi}$ to $S_{\phi}$ until none of the remaining elements in $B_{\phi}$ can be included without violating the $d$-knapsack constraint (Lines 34–37). After the post-processing, it also returns the candidate with the maximum utility among the candidates in $\mathcal{H}(x_{1})$ or $\mathcal{H}(x_{2})$ as the final solution $S_{t}$.

5.2 Theoretical Analysis

Next, we analyze the approximation ratio and complexity of $\text{KW}^{+}$. We first prove the properties of the checkpoints in SubKnapChk (Lemma 3). Based on the properties, we prove the approximation ratio of $\text{KW}^{+}$ (Theorem 3). Finally, we analyze the number of checkpoints in SubKnapChk, calculate the cost of buffer maintenance and post-processing for $\text{KW}^{+}$, and acquire the complexity of $\text{KW}^{+}$.

First of all, we prove the properties of the checkpoints in SubKnapChk.

Lemma 3**.**

Given a parameter $\beta\in(0,1)$, each checkpoint $x_{i}\in X_{t}$ where $i\in[1,s]$ maintained by SubKnapChk at time $t$ satisfies one of the following properties:

if $f[x_{i+1},t]\geq(1-\beta)f[x_{i},t]$, $f[x_{i+2},t]<(1-\beta)f[x_{i},t]$ or $x_{i+1}=x_{s}$. 2. 2.

if $x_{i+1}\neq x_{i}+1$ and $f[x_{i+1},t]<(1-\beta)f[x_{i},t]$, there exists some $t^{\prime}<t$ such that $f[x_{i+1},t^{\prime}]\geq(1-\beta)f[x_{i},t^{\prime}]$. 3. 3.

$x_{i+1}=x_{i}+1$* and $f[x_{i+1},t]<(1-\beta)f[x_{i},t]$.*

Proof.

We prove the lemma by induction on $t$. As the base case, we first check the condition when $t=2$ and $X_{2}=\langle x_{1}=1,x_{2}=2\rangle$. Then, Property (1) holds if $f[x_{2},2]\geq(1-\beta)f[x_{1},2]$; otherwise, Property (3) holds.

Next, we assume Lemma 3 holds at time $t$ and show that it still holds after performing Lines 5–17 of Algorithm 3 at time $t+1$. Let $x_{i}$ be a checkpoint that is created before $t+1$ and not deleted during the maintenance at time $t+1$ and $x_{i+1}$ be the checkpoint next to $x_{i}$ at time $t$. We discuss all possible cases during the maintenance at time $t+1$.

Case 1. $x_{i+1}\neq x_{i}+1$ and $x_{i+1}$ is deleted from SubKnapChk at time $t+1$. In this case, we have $f[x_{i+2},t+1]\geq(1-\beta)f[x_{i},t+1]$ (Line 15 of Algorithm 3). As $x_{i+2}$ becomes the successor of $x_{i}$ at time $t+1$, Property (1) holds.

Case 2. $x_{i+1}\neq x_{i}+1$ and $x_{i+1}$ is not deleted from SubKnapChk at time $t+1$. In this case, we consider $x_{i+1}$ becomes the successor of $x_{i}$ at some time $t^{\prime}\leq t$. Then, it must hold that $f[x_{i+1},t^{\prime}]\geq(1-\beta)f[x_{i},t^{\prime}]$. Since $x_{i+1}$ is not deleted at time $t+1$, either Property (1) (if $f[x_{i+1},t+1]\geq(1-\beta)f[x_{i},t+1]$) or Property (2) (if $f[x_{i+1},t+1]<(1-\beta)f[x_{i},t+1]$) holds.

Case 3. $x_{i+1}=x_{i}+1$. No matter whether $x_{i+1}$ is deleted at time $t+1$, Property (1) holds if $f[x_{i+1},t+1]\geq(1-\beta)f[x_{i},t+1]$; otherwise, Property (3) holds.

We show that the properties of SubKnapChk still hold at time $t+1$ in all possible cases and conclude the proof. ∎

Given the properties of SubKnapChk, we can analyze the approximation ratio of $S_{t}$ returned by Algorithm 3 for SMDK w.r.t. $A_{t}$.

Theorem 3**.**

The solution $S_{t}$ returned by Algorithm 3 satisfies that $f(S_{t})\geq\frac{1-\varepsilon^{\prime}}{2(1+d)}\cdot\mathtt{OPT}_{t}$ at any time $t$ where $\varepsilon^{\prime}=\varepsilon+\beta$.

Proof.

We consider the first two checkpoints $x_{1}$ and $x_{2}$ of SubKnapChk at time $t$ and assume that post-processing does not change the solution $S_{t}$. If $t\leq W$, $x_{1}=1$ does not expire and $\mathcal{H}(x_{1})$ are maintained over $A_{t}=\langle v_{1},\ldots,v_{t}\rangle$. Thus, $f(S_{t})=f[x_{1},t]\geq\frac{1-\varepsilon}{1+d}\mathtt{OPT}_{t}$ for $t\leq W$ by Theorem 1. Next, we consider $t>W$ and $x_{2}=x_{1}+1$. In this case, $x_{1}$ expires and $x_{2}$ corresponds to the starting point of $A_{t}$. Similarly, $f(S_{t})=f[x_{2},t]\geq\frac{1-\varepsilon}{1+d}\mathtt{OPT}_{t}$.

Subsequently, we consider other cases for $t>W$. We use $\mathtt{OPT}^{x}_{y}$ to denote the optimal utility of SMDK w.r.t. the elements $\langle v_{x},\ldots,v_{y}\rangle$.

Case 1. If $f[x_{2},t]\geq(1-\beta)f[x_{1},t]$, $f(S_{t})=f[x_{2},t]\geq(1-\beta)f[x_{1},t]$. By Theorem 1, $f[x_{1},t]\geq\frac{1-\varepsilon}{1+d}\mathtt{OPT}^{x_{1}}_{t}$. As $x_{1}<t^{\prime}$, we have $A_{t}\subset\langle v_{x_{1}},\ldots,v_{t}\rangle$ and $\mathtt{OPT}_{t}\leq\mathtt{OPT}^{x_{1}}_{t}$. Finally, we have $f(S_{t})\geq\frac{(1-\beta)(1-\varepsilon)}{1+d}\mathtt{OPT}_{t}$.

Case 2. If $f[x_{2},t]<(1-\beta)f[x_{1},t]$, we have $f[x_{2},t^{\prime}]\geq(1-\beta)f[x_{1},t^{\prime}]$ for some $t^{\prime}<t$. Let $S^{*}_{x_{1},t}$ denote the optimal solution for $\langle v_{x_{1}},\ldots,v_{t}\rangle$. We can split $S^{*}_{x_{1},t}$ into two subsets $S_{1}$ and $S_{2}$, where $S_{1}=\{v_{i}|v_{i}\in S^{*}_{x_{1},t}\wedge i\in[x_{1},t^{\prime}]\}$ and $S_{2}=\{v_{i}|v_{i}\in S^{*}_{x_{1},t}\wedge i\in[x_{2},t]\}$. Let $\mathtt{OPT}_{1}=f(S_{1})$ and $\mathtt{OPT}_{2}=f(S_{2})$. For $S^{*}_{x_{1},t}=S_{1}\cup S_{2}$ and the submodularity of $f(\cdot)$, $\mathtt{OPT}^{x_{1}}_{t}\leq\mathtt{OPT}_{1}+\mathtt{OPT}_{2}$. Then, as $S_{1}\in\xi$ and $S_{2}\in\xi$, it holds that $\mathtt{OPT}_{1}\leq\mathtt{OPT}^{x_{1}}_{t^{\prime}}$ and $\mathtt{OPT}_{2}\leq\mathtt{OPT}^{x_{2}}_{t}$. In addition, for any $t_{1}<t_{2}$, the solution returned by KnapStream satisfies that $f[x,t_{1}]\leq f[x,t_{2}]$. As $t>t^{\prime}$, we have:

[TABLE]

We also have:

[TABLE]

Adding the above two inequalities, we prove:

[TABLE]

Finally, because the post-processing procedure must not decrease the utility of any candidate, Equation 4 still holds after post-processing. Thus, we conclude the proof by replacing $\lambda$ and $\varepsilon$ with $\varepsilon^{\prime}$ in Equation 4. ∎

The Complexity of $\text{KW}^{+}$. According to Lemma 3, either $f[x_{i+1},t]$ or $f[x_{i+2},t]$ is less that $(1-\beta)f[x_{i},t]$ at any time $t$. Given $\theta=\frac{f[x_{1},t]}{f[x_{s},t]}$, the number of checkpoints in SubKnapChk is at most $\lceil\frac{2\log\theta}{\log(1-\beta)^{-1}}\rceil$. Therefore, the number of checkpoints is $\mathcal{O}(\frac{\log\theta}{\beta})$. $\text{KW}^{+}$ performs $\mathcal{O}(\frac{\log\theta\cdot\log(d\cdot\gamma^{-1})}{\varepsilon^{\prime 2}})$ function calls to update the candidates in the checkpoints for one element and stores at most $\mathcal{O}(\frac{\log\theta\cdot\log(d\cdot\gamma^{-1})}{\gamma\cdot\varepsilon^{\prime 2}})$ elements within the candidates. In practice, the buffer of each candidate is implemented by a min-heap and the buffer size $\eta=\mathcal{O}(\gamma^{-1})$. The complexity of adding an element to the buffer is $\mathcal{O}(\log\gamma^{-1})$ and dropping elements from the buffer is $\mathcal{O}(\gamma^{-1})$. Thus, the amortized computational cost for buffer maintenance is $\mathcal{O}(\frac{\log\theta\cdot\log(d\cdot\gamma^{-1})}{\varepsilon^{\prime 2}})$ and the total number of elements in buffers is $\mathcal{O}(\frac{\log\theta\cdot\log(d\cdot\gamma^{-1})}{\gamma\cdot\varepsilon^{\prime 2}})$. The post-processing for one candidate handles $\mathcal{O}(\gamma^{-1})$ elements and runs at most $\gamma^{-1}$ iterations. Therefore, the post-processing requires $\mathcal{O}(\frac{\log(d\cdot\gamma^{-1})}{\gamma^{2}\cdot\varepsilon^{\prime}})$ function calls. Generally, $\text{KW}^{+}$ runs in \mathcal{O}\big{(}\frac{\log(d\cdot\gamma^{-1})}{\varepsilon^{\prime}}\cdot(\gamma^{-2}+\frac{\log\theta}{\varepsilon^{\prime}})\big{)} time to process one element and stores $\mathcal{O}(\frac{\log\theta\cdot\log(d\cdot\gamma^{-1})}{\gamma\cdot\varepsilon^{\prime 2}})$ elements in total.

5.3 Discussion

In practice, it is no need to update the solution for every arrival element. The update is often performed in a batch manner. Specifically, we consider the sliding window receives $T$ new elements while the earliest $T$ elements become expired at time $t$. Both KW and $\text{KW}^{+}$ can handle the scenario with trivial adaptations. For KW, it also stores the active elements in $A_{t}$ and creates a checkpoint for every $L$ elements. The only difference is that the interval $L$ becomes $\sqrt{W\cdot T}$ while the number of checkpoints $s$ decreases to $\sqrt{\frac{W}{T}}$. For $\text{KW}^{+}$, it creates one checkpoint at each time $t$ and updates existing checkpoints by processing a batch of elements from $v_{t-T+1}$ to $v_{t}$ collectively. In this way, the total number of checkpoints created is $\lceil\frac{W}{T}\rceil$. The number of checkpoints in SubKnapChk is determined by the utilities and thus is not affected. In addition, any other theoretical results, the buffer maintenance, and the post-processing procedure are also not affected by these adaptations.

6 Experiments

In this section, we report our experimental results for two RSS applications (as presented in Section 3) in real-world datasets. First, we introduce the experimental setup in Section 6.1. Then, we evaluate the effectiveness and efficiency of our proposed frameworks compared with several baselines in Section 6.2.

6.1 Experimental Setup

Datasets. Two real-world datasets are used in our experiments. First, we use the Twitter dataset for social stream summarization (see Section 3.1). It is collected via the streaming API222http://twitter4j.org/en/index.html and contains $18,770,231$ tweets and $8,071,484$ words. The average number of words in each tweet is $\overline{l}=4.8$ and the average number of followers of each user is $\overline{fl}=521.4$. In the experiments, we feed each tweet to the compared approaches one by one in ascending order of timestamp. Second, we use the Yahoo! Webscope dataset333http://webscope.sandbox.yahoo.com for active set selection (see Section 3.2). It consists of $45,811,883$ user visits from the Featured Tab of the Today module on the Yahoo! front page. Each user visit is a $5$-dimensional feature vector. We set $h=0.75$ and $\sigma=1$ in Equation 3 following [2]. The costs are generated from a uniform distribution $\mathcal{U}(0.02,0.08)$. In the experiments, we feed all user visits to the compared approaches one by one in the same order.

Additional constraints. To evaluate the compared approaches with varying the dimension of knapsacks, i.e., $d$, we generate additional constraints by assigning random costs to each element in both datasets. Specifically, we generate a $5$-dimensional cost vector $\mathbf{c}(v)=\{c_{1}(v),\ldots,c_{5}(v)\}$ for each element $v$. And each cost is generated independently from a uniform distribution $\mathcal{U}(0.02,0.08)$. We set $d$ to range from $1$ to $5$ in the experiments and use the first $d$ dimensions of $\mathbf{c}(v)$ for the $d$-knapsack constraint.

Compared Approaches. The approaches compared in our experiments are listed as follows.

• •

CostEffectGreedy (CEG). We implement the CostEffectGreedy algorithm [17] as the batch baseline. Since CEG is designed for submodular maximization with a $1$-knapsack constraint, we slightly adapt it for SMDK: the cost-effectiveness of $v$ w.r.t. $S$ is computed by $\frac{\Delta_{f}(v|S)}{\delta(v)}$ where $\delta(v)=\max_{j\in[d]}c(v)$. To work in the sliding window model, it stores the active elements in $A_{t}$ and recomputes the solution from scratch for each window slide.

• •

Streaming (STR). We implement the state-of-the-art append-only streaming algorithm [38] for SMDK as a baseline. To work in the sliding window model, it also stores the active elements in $A_{t}$ and recomputes the solution from scratch for each window slide.

• •

Window (WIN). We implement the state-of-the-art algorithm for submodular maximization over sliding windows [6] as a baseline. Since it only works with one cardinality constraint, we cast the $d$-knapsack constraint to the cardinality constraint by setting the budget $k=\frac{1}{\gamma}$ where $\gamma$ is the average cost of elements. When maintaining the solutions over sliding windows, it only considers the marginal gains of elements and treats the cost of any element as $1$.

• •

KnapWindow (KW). We implement the KnapWindow framework in Section 4.

• •

KnapWindowPlus ($\text{KW}^{+}$). We implement the KnapWindowPlus framework in Section 5. We set $\alpha=0.5$ and $\eta=20$ for buffer maintenance.

Parameters. The parameters tested in our experiments are listed in Table 2 with default values in bold. $d$ is the dimension of the knapsack constraint. We use $d=3$ for social stream summarization and $d=1$ for active set selection by default as introduced in Section 3; $c$ is the average cost of each element. For the Twitter dataset, we set $k=\frac{1}{c}$ to assign the costs $c_{1}(v),c_{2}(v),c_{3}(v)$ accordingly as introduced in Section 3.1. For the Yahoo! Webscope dataset, the average of generated costs is $c=0.05$. We scale the costs linearly in the experiments for varying $c$. $W$ is the size of the sliding window. We set the number of elements for each window slide to $T=0.01\%\cdot W$. The interval for neighboring checkpoints in KW is $L=\sqrt{W\cdot T}=1\%\cdot W$ (Section 5.3). $\lambda$ is the parameter used in KW, $\text{KW}^{+}$, STR, and WIN for the balance between the number of candidates maintained for processing append-only streams and solution quality. $\beta$ is the parameter for $\text{KW}^{+}$ to balance between the number of checkpoints and solution quality.

Metrics. We consider the following metrics to evaluate the compared approaches.

• •

CPU time is the average CPU time of an approach to process one window slide. It is used to measure the efficiency of compared approaches.

• •

Utility is the average utility value of the solution returned by an approach for each window. It evaluates the solution quality of compared approaches.

• •

#checkpoints and #elements are the average numbers of checkpoints and elements maintained by $\text{KW}^{+}$, which are used to measure its space usage.

Experimental Environment. All the above approaches are implemented in Java 8 and the experiments are conducted on a server running Ubuntu 16.04 with a 1.9GHz Intel Xeon E7-4820 processor and 128 GB memory.

6.2 Experimental Results

Overall Results. In Figure 4, we present the CPU time and utilities of compared approaches in the default setting. Although CEG achieves the best utilities, it takes around 10s to process each window slide, which is far lower than the rates of real-world data streams. KW and $\text{KW}^{+}$ run over two and three orders of magnitude faster than CEG respectively and can process each window slide within 100ms. Meanwhile, the utilities of the solutions provided by KW and $\text{KW}^{+}$ are about $85\%$ of those of CEG. Furthermore, $\text{KW}^{+}$ significantly improves the efficiency upon KW, achieving speedups of at least $6$x in both datasets. Compared with STR, KW and $\text{KW}^{+}$ run dozens of times faster while providing solutions with similar utilities. Finally, we observe WIN runs faster than other approaches but shows obviously inferior solution quality. This is because WIN treats the costs of any element equally and only considers marginal utility gains when adding an element. As a result, the solutions of WIN contain fewer elements than other approaches, which leads to both higher efficiency and worse solution quality.

In Figure 5, we present the utilities of compared approaches from time $t=W$ to the end of the stream $t=n$. The solutions returned by CEG achieve the highest utilities all the time. the solution utilities of KW, $\text{KW}^{+}$, and STR fluctuate over time and are generally close to each other. But remember that $\text{KW}^{+}$ takes much less CPU time than KW while KW runs significantly faster than STR (as illustrated in Figure 4). Also as expected, the solution quality of WIN cannot match any other approaches.

To sum up, $\text{KW}^{+}$ achieves the best balance between efficiency and solution quality: compared with CEG, it runs more than three orders of magnitude faster while providing solutions with 85% average utility; it has much higher efficiency than KW and STR but achieves equivalent solution quality; it significantly improves the solution quality upon WIN at a little expense of efficiency.

Effect of $d$. The CPU time and utilities of compared approaches with varying $d$ are shown in Figure 6. The CPU time of CEG decreases when $d$ increases. This is because the average solution size becomes smaller when there are more constraints. The CPU time of KW shows different trends in both datasets: it decreases in the Yahoo! Webscope dataset but keeps steady in the Twitter dataset when $d$ becomes larger. There are two observations behind such trends: First, since KS maintains the candidates for estimations from $m$ to $M(1+d)$ (see Algorithm 1), a KS instance maintains more candidates with increasing $d$. Second, the average solution size decreases with $d$. In the Twitter dataset, the extra costs for maintaining more candidates cancel out the benefits of smaller solutions and thus the overall CPU time keeps steady. However, in the Yahoo! Webscope dataset, the time complexity of evaluating IVM in Equation 3 for a set $S$ is $\mathcal{O}(|S|^{3})$. As a result, the CPU time for each IVM evaluation is very sensitive to $|S|$. Although more candidates are maintained, the overall CPU time of KW still becomes much lower. The CPU time of $\text{KW}^{+}$ shows a similar trend to KW in the Twitter dataset. But it keeps steady with increasing $d$ in the Yahoo! Webscope dataset. The reason behind such an observation is, although the CPU time to update the checkpoints decreases, the post-processing takes longer time when $d$ increases. The utilities of all compared approaches decrease when $d$ increases because of smaller solution sizes. Compared with KW and STR, $\text{KW}^{+}$ shows slightly better solution quality for a larger $d$ due to the benefits of post-processing. In addition, the ratios between the utilities of the solutions of STR, KW, and $\text{KW}^{+}$ and those of CEG are $84\%$–$90\%$ and remain stable for different $d$.

Robustness against $\lambda$ and $\beta$. The experimental results of compared approaches with varying parameters $\lambda$ are shown in Figure 7. For all compared approaches except CEG, the CPU time obviously drops with increasing $\lambda$. This is because the number of candidates is inversely correlated to $\lambda$. However, we observe that their utilities are rather robust against $\lambda$ and only slightly decrease for a larger $\lambda$. The utility of $\text{KW}^{+}$ in the Yahoo! Webscope dataset even increases when $\lambda=0.15$ thanks to the post-processing.

The experimental results of compared approaches with varying parameters $\beta$ are shown in Figure 8. Because the number of checkpoints and KS instances in SubKnapChk is inversely correlated to $\beta$, the CPU time of $\text{KW}^{+}$ decreases when $\beta$ increases. However, the robustness of $\text{KW}^{+}$ against $\beta$ is worse than its robustness against $\lambda$. The utilities show drastic drops when $\beta=0.2$ or $0.25$. As the intervals between the first two checkpoints increase with $\beta$, the errors of using the results from the second checkpoint as the solutions inevitably increase. Considering the results, we advise using a small $\beta$ so that $\text{KW}^{+}$ can achieve good solution quality.

Scalability. In Figure 9, we present the CPU time of compared approaches with varying $W$ and $c$. The CPU time to process each window slide increases with $W$. This is because the number of elements processed for each window slide is set to $0.01\%\cdot W$ which increases linearly with $W$. For all compared approaches, it takes a longer CPU time when $c$ decreases because the solution size is inversely proportional to $c$. In the Yahoo! Webscope dataset, the CPU time increases drastically when $c$ decreases because the time complexity of the IVM function evaluation is $\mathcal{O}(|S|^{3})$. Thus, all compared approaches spend much more CPU time for each evaluation of $f(S)$ when the solution size grows. Nevertheless, the CPU time of $\text{KW}^{+}$ and that of KW are within $100$ms and $1$s respectively in all parameter settings.

We list the number of checkpoints and the number of elements in both candidates and buffers maintained by $\text{KW}^{+}$ with varying $W$ and $c$ in Table 3. First, because the number of checkpoints and the number of elements are independent of $W$ and bounded by the ratio of the utilities of the solutions provided by the first and last checkpoints, both metrics hardly increases with $W$. In addition, the number of elements in $\text{KW}^{+}$ increases when $c$ decreases because each candidate maintains more elements. Generally, $\text{KW}^{+}$ only stores several thousand elements when $W$ ranges from $100$k to $500$k. Taking $W=500$k as an example, $\text{KW}^{+}$ merely stores $0.7\%$ of the active elements. Therefore, the space usage of $\text{KW}^{+}$ is much smaller than KW, CEG, and STR, which need to store the entire active window. Furthermore, the number of elements maintained by $\text{KW}^{+}$ does not increase with the window size $W$ because the space complexity of $\text{KW}^{+}$ is independent of $W$. Hence, $\text{KW}^{+}$ is scalable for large window sizes.

7 Related Work

Representative subset selection (RSS) is an important tool to draw insights from massive datasets. Existing RSS techniques can be categorized into four classes based on the utility functions used to evaluate the representativeness: (1) coverage-based RSS [29, 22, 23, 32, 35, 37, 40]; (2) entropy-based RSS [2, 10, 36, 6]; (3) clustering-based RSS [2, 10, 24, 25]; (4) diversity-aware RSS [22, 23, 26]. Coverage-based approaches treat RSS as the maximum coverage problem [29] and its variants, e.g., budgeted coverage [22, 23], weighted coverage [32, 40], and probabilistic coverage [37]. They consider all information in a dataset as a collection of information units. The objective of RSS is to select a subset of elements so as to maximally cover the information units in the source dataset. Entropy-based RSS [2, 10, 36, 6] (a.k.a. active set selection) aims to select a subset of elements with the highest information entropy. Active set selection is considered as a powerful tool for large-scale nonparametric learning [2, 10]. Clustering-based RSS [2, 10, 24, 25] (a.k.a. exemplar clustering) selects a subset of elements such that the average distance from the remaining elements in the dataset to their nearest neighbor in the selected subset is minimized. Diversity-aware RSS [22, 23, 26] integrates a coverage/clustering based utility function with a diversity function to avoid including highly similar elements into the selected subset. Generally, the utility functions used in the aforementioned RSS problems are all submodular because the representativeness naturally satisfies the “diminishing returns” property. But most of them [22, 23, 35, 37, 40, 36, 24, 25, 26] can only work in the batch setting and are very inefficient to process data streams.

Recently, we have witnessed the growth of RSS studies in the data stream model. RSS in append-only streams where new elements arrive continuously but old ones never expire is studied in [29, 2, 10, 32]. Mirzasoleiman et al. [27] further propose a method for deletion-robust RSS where a limited number of old elements can be deleted from the stream. However, these techniques neither support general constraints beyond cardinality nor consider the recency of selected subsets. In many scenarios, data streams are highly dynamic and evolve over time. Therefore, recent elements are more important and interesting than earlier ones. The sliding window [5] model is widely adopted in many data-driven applications [33, 34] to capture the recency constraint. RSS over sliding windows is still largely unexplored yet and, to the best of our knowledge, there is only one existing method [6] for dynamic RSS over sliding windows. But it is specific for the cardinality constraint. In this paper, we propose more general frameworks for RSS than any existing ones, which work with various submodular utility functions, support $d$-knapsack constraints, and maintain the representatives over sliding windows.

Submodular maximization (SM) has been extensively studied in recent years. Due to its theoretical consequences, SM is seen as a “silver bullet” for many different applications [38, 17, 34, 20, 21]. Here, we focus on reviewing existing literature on SM that is closely related to our paper: SMDK and SM in data streams. Sviridenko [31] and Kulik et al. [14] first propose approximation algorithms for SM subject to $1$-knapsack and $d$-knapsack constraints respectively. Both algorithms have high-order polynomial time complexity and are not scalable to massive datasets. More efficient algorithms for SM subject to $1$-knapsack constraints are proposed in [17, 23] and [1] respectively. These algorithms cannot be applied to SMDK directly. Badanidiyuru et al. [2] and Kumar et al. [15] propose the algorithms for SM with cardinality constraints in append-only streams with sublinear time complexity. Then, Huang et al. [12] propose an algorithm for SM in append-only streams with $1$-knapsack constraints. Yu et al. [38] propose an algorithm for SMDK in append-only streams. More recently, there are a few attempts at SM over sliding windows. Epasto et al. [6] propose an algorithm for SM over sliding windows with cardinality constraints. To the best of our knowledge, there is no existing literature on SMDK over sliding windows yet.

We compare the above SM algorithms theoretically in Table 4. We present their data models, supported constraints, approximation factors, and time complexities respectively. According to the results, our contributions in this paper are two-fold: (1) KS improves the approximation factor of SMDK in append-only streams from $\frac{1}{1+2d}-\varepsilon$ to $\frac{1-\varepsilon}{1+d}$; (2) KW and $\text{KW}^{+}$ are among the first algorithms for SMDK in the sliding window model.

8 Conclusion

In this paper, we studied the representative subset selection (RSS) problem in data streams. First of all, we formulated dynamic RSS as maximizing a monotone submodular function subject to a $d$-knapsack constraint (SMDK) over sliding windows. We then devised the KW framework for this problem. Theoretically, KW provided solutions for SMDK over sliding windows with an approximation factor of $\frac{1-\varepsilon}{1+d}$. Furthermore, we proposed a more efficient $\frac{1-\varepsilon^{\prime}}{2+2d}$-approximation $\text{KW}^{+}$ framework for SMDK over sliding windows. The experimental results demonstrated that KW and $\text{KW}^{+}$ run orders of magnitude faster than the batch baseline while preserving high-quality solutions.