Streaming Non-monotone Submodular Maximization: Personalized Video Summarization on the Fly

TL;DR

This paper introduces a novel streaming algorithm for non-monotone submodular maximization, enabling real-time, personalized video summarization with high efficiency and comparable quality to existing methods.

Contribution

It presents the first efficient single-pass streaming algorithm for non-monotone submodular maximization with provable approximation guarantees under complex constraints.

Findings

Runs over 1700 times faster than previous methods

Maintains similar summarization quality

Effective for real-time personalized video summarization

Abstract

The need for real time analysis of rapidly producing data streams (e.g., video and image streams) motivated the design of streaming algorithms that can efficiently extract and summarize useful information from massive data "on the fly". Such problems can often be reduced to maximizing a submodular set function subject to various constraints. While efficient streaming methods have been recently developed for monotone submodular maximization, in a wide range of applications, such as video summarization, the underlying utility function is non-monotone, and there are often various constraints imposed on the optimization problem to consider privacy or personalization. We develop the first efficient single pass streaming algorithm, Streaming Local Search, that for any streaming monotone submodular maximization algorithm with approximation guarantee $\alpha$ under a collection of independence systems ${\cal I}$, provides a constant $1/\big(1+2/\sqrt{\alpha}+1/\alpha +2d(1+\sqrt{\alpha})\big)$ approximation guarantee for maximizing a non-monotone submodular function under the intersection of ${\cal I}$ and $d$ knapsack constraints. Our experiments show that for video summarization, our method runs more than 1700 times faster than previous work, while maintaining practically the same performance.

Tables

Table 1. Table 1: Performance of various video summarization methods with segment size 10 on YouTube and OVP datasets, measured by F-Score (F), Precision (P), and Recall (R).

		Alg. of (?)${}^{\text{(centralized)}}$		Fantom ${}^{\text{(centralized)}}$		Streaming Local Search
		Linear	N. Nets	Linear	N. Nets	Linear	N. Nets
YouTube	F	57.8$\pm$0.5	60.3$\pm$0.5	57.7$\pm$0.5	60.3$\pm$0.5	58.3$\pm$0.5	59.8$\pm$0.5
	P	54.2$\pm$0.7	59.4$\pm$0.6	54.1$\pm$0.5	59.1$\pm$0.6	55.2$\pm$0.5	58.6$\pm$0.6
	R	69.8$\pm$0.5	64.9$\pm$0.5	70.1$\pm$0.5	64.7$\pm$0.5	70.1$\pm$0.5	64.2$\pm$0.5
OVP	F	75.5$\pm$0.4	77.7$\pm$0.4	75.5$\pm$0.3	78.0$\pm$0.5	74.6$\pm$0.2	75.6$\pm$0.5
	P	77.5$\pm$0.5	75.0$\pm$0.5	77.4$\pm$0.3	75.1$\pm$0.7	76.7$\pm$0.2	71.8$\pm$0.7
	R	78.4$\pm$0.5	87.2$\pm$0.3	78.4$\pm$0.3	88.6$\pm$0.2	76.5$\pm$0.3	86.5$\pm$0.2

Equations

$$f(S\cup \{e\})-f(S)\ge f(T\cup \{e\})-f(T).$$

$$S^{\ast }=\text{mbox}{argmax}_{S\in \zeta }f(S).$$

$$\mathcal{P}(Y=S)=\frac{\det (L_S)}{\det (I+L)},$$

$$f(S)\ge \frac{1}{(1+1/\sqrt{\alpha }{)}^2}\text{textsc}OPT,$$

$$f(S)\ge \frac{1-ϵ}{(1+1/\sqrt{\alpha })(1+2d\sqrt{\alpha }+1/\sqrt{\alpha })}\text{textsc}OPT,$$

$$f(S)\ge \frac{1-ϵ}{(1+1/\sqrt{\alpha })(1+2d\sqrt{\alpha }+1/\sqrt{\alpha })}(\text{textsc}OPT-\epsilon ).$$

$$f(S_i)\ge \alpha f(S_i\cup C_i),$$

$$f(S_i^{\prime })\ge \beta f(S_i\cap C_i),$$

$$\mathbb{E}[f^{\prime }(S)]$$

$$f(C_i)+f(C^{\ast }\cap ({\cup }_{j=1}^{i-1}{S}_j))=f(C_i)+f({\cup }_{j=1}^{i-1}(C^{\ast }\cap S_j))=f(C^{\ast }).$$

$$(r-1)f(C^{\ast })$$

$$\displaystyle\leq\sum_{i=1}^{r}\left[f(S_{i}\cup C_{i})+f\big{(}\cup_{j=1}^{i-1}(C^{*}\cap S_{j})\big{)}\right]$$

$$\displaystyle\leq\sum_{i=1}^{r}\bigg{[}f(S_{i}\cup C_{i})+\sum_{j=1}^{i-1}f(C^{*}\cap S_{j})\bigg{]}$$

$$\displaystyle\leq\sum_{i=1}^{r}\bigg{[}\frac{1}{\alpha}f(S_{i})+\frac{1}{\beta}\sum_{j=1}^{i-1}f(S^{\prime}_{j})\bigg{]}$$

$$\displaystyle\leq\sum_{i=1}^{r}\bigg{[}\frac{1}{\alpha}f(S)+\frac{1}{\beta}\sum_{j=1}^{i-1}f(S)\bigg{]}$$

$$=\left(\frac{r}{\alpha }+\frac{r(r-1)}{2\beta }\right)f(S).$$

$$f(S)\ge \frac{r-1}{r/\alpha +r(r-1)/2\beta }f(C^{\ast })$$

$$f(S)$$

$$=\frac{\sqrt{2\beta /\alpha }}{(\sqrt{\frac{2\beta }{\alpha }}+1)(\frac{1}{\alpha }+\frac{\sqrt{2\beta /\alpha }}{2\beta })}f(C^{\ast })$$

$$=\frac{\sqrt{2\beta }}{(\sqrt{2\beta }+1/\sqrt{\alpha })(1/\sqrt{\alpha }+1/\sqrt{2\beta })}f(C^{\ast })$$

$$=\frac{\sqrt{2\beta }}{(1/\sqrt{\alpha }+1/\sqrt{2\beta }{)}^2}f(C^{\ast })$$

$$f(S)\ge \frac{1}{(1/\sqrt{\alpha }+1{)}^2}f(C^{\ast })$$

$$f(S)\ge \frac{1}{(2\sqrt{p}+1{)}^2}f(C^{\ast })$$

$${\rho }^{\ast }=\frac{2}{\left(\frac{1}{\sqrt{\alpha }}+\frac{1}{\sqrt{\beta }}\right)\left(\frac{1}{\sqrt{\alpha }}+2d\sqrt{\alpha }+\frac{1}{\sqrt{\beta }}\right)}f(C^{\ast }).$$

$$\frac{2m}{\left(\frac{1}{\sqrt{\alpha }}+\frac{1}{\sqrt{\beta }}\right)\left(\frac{1}{\sqrt{\alpha }}+2d\sqrt{\alpha }+\frac{1}{\sqrt{\beta }}\right)}\le {\rho }^{\ast }\le \frac{2mk}{\left(\frac{1}{\sqrt{\alpha }}+\frac{1}{\sqrt{\beta }}\right)\left(\frac{1}{\sqrt{\alpha }}+2d\sqrt{\alpha }+\frac{1}{\sqrt{\beta }}\right)}.$$

$$\rho \le {\rho }^{\ast }\le (1+ϵ)\rho .$$

$$\displaystyle f(S_{\rho})\geq\sum_{e\in S_{\rho}}\big{(}\rho\sum_{j=1}^{d}c_{je}\big{)}>{\rho}.$$

$$\max \{f(T_{\rho }),f(\{e_f\})\}\ge \frac{\rho }{2}.$$

$$f_{S_{\rho }}(C_{<\rho }^{\ast })\le \sum _{e\in C_{<\rho }}\rho \sum _{j=1}^d{c}_{je}=\rho \sum _{j=1}^d\sum _{e\in C_{<\rho }}{c}_{je}$$

$$f_{S_{\rho }}(C_{<\rho }^{\ast })\le d\rho \le \rho \sum _{j=1}^d\sum _{e\in C_{<\rho }}{c}_{je}\le d\rho \le d{\rho }^{\ast }$$

$$f(S_{\rho })\ge \alpha f(S_{\rho }\cup C_{\ge \rho }^{\ast }),$$

$$f_{S_{\rho }}(C_{\ge \rho }^{\ast })=$$

$$f(S_{\rho }\cup C^{\ast })-f(S_{\rho })\le$$

$$f(S_{\rho })\ge \alpha f(S_{\rho }\cup C^{\ast })-\alpha d\rho .$$

Full text

Introduction

Data summarization–the task of efficiently extracting a representative subset of manageable size from a large dataset–has become an important goal in machine learning and information retrieval. Submodular maximization has recently been explored as a natural abstraction for many data summarization tasks, including image summarization (?), scene summarization (?), document and corpus summarization (?), active set selection in non-parametric learning (?) and training data compression (?). Submodularity is an intuitive notion of diminishing returns, stating that selecting any given element earlier helps more than selecting it later. Given a set of constraints on the desired summary, and a (pre-designed or learned) submodular utility function $f$ that quantifies the representativeness $f(S)$ of a subset $S$ of items, data summarization can be naturally reduced to a constrained submodular optimization problem.

In this paper, we are motivated by applications of non-monotone submodular maximization. In particular, we consider video summarization in a streaming setting, where video frames are produced at a fast pace, and we want to keep an updated summary of the video so far, with little or no memory overhead. This has important applications e.g. in surveillance cameras, wearable cameras, and astro video cameras, which generate data at too rapid a pace to efficiently analyze and store it in main memory. The same framework can be applied more generally in many settings where we need to extract a small subset of data from a large stream to train or update a machine learning model. At the same time, various constraints may be imposed by the underlying summarization application. These may range from a simple limit on the size of the summary to more complex restrictions such as focusing on particular individuals or objects, or excluding them from the summary. These requirements often arise in real-world scenarios to consider privacy (e.g. in case of surveillance cameras) or personalization (according to users’ interests).

In machine learning, Determinantal Point Processes (DPP) have been proposed as computationally efficient methods for selecting a diverse subset from a ground set of items (?). They have recently shown great success for video summarization (?), document summarization (?) and information retrieval (?). While finding the most likely configuration (MAP) is NP-hard, the DPP probability is a log-submodular function, and submodular optimization techniques can be used to find a near-optimal solution. In general, the above submodular function is very non-monotone, and we need techniques for maximizing a non-monotone submodular function in the streaming setting. Although efficient streaming methods have been recently developed for maximizing a monotone submodular function $f$ with a variety of constraints, there is no effective streaming solution for non-monotone submodular maximization under general types of constraints.

In this work, we provide Streaming Local Search, the first single pass streaming algorithm for non-monotone submodular function maximization, subject to the intersection of a collection of independence systems ${\cal I}$ and $d$ knapsack constraints. Our approach builds on local search, a widely used technique for maximizing non-monotone submodular functions in a batch mode. Local search, however, needs multiple passes over the input, and hence does not directly extend to the streaming setting, where we are only allowed to make a single pass over the data. This work provides a general framework within which we can use any streaming monotone submodular maximization algorithm, IndStream, with approximation guarantee $\alpha$ under a collection of independence systems ${\cal I}$. For any such monotone algorithm, Streaming Local Search provides a constant 1/\big{(}1\!+\!2/\sqrt{\alpha}\!+\!1/\alpha+2d(1\!+\!\sqrt{\alpha})\big{)} approximation guarantee for maximizing a non-monotone submodular function under the intersection of ${\cal I}$ and $d$ knapsack constraints. Furthermore, Streaming Local Search needs a memory and update time that is larger than IndStream with a factor of $O(\log(k)/\sqrt{\alpha})$, where $k$ is the size of the largest feasible solution. Using parallel computation, the increase in the update time can be reduced to $O(1/\sqrt{\alpha})$, making our approach an appealing solution in real-time scenarios. We show that for video summarization, our algorithm leads to streaming solutions that provide competitive utility when compared with those obtained via centralized methods, at a small fraction of the computational cost, i.e. more than 1700 times faster.

Related Work

Video summarization aims to retain diverse and representative frames according to criteria such as representativeness, diversity, interestingness, or frame importance (?; ?; ?). This often requires hand-crafting to combine the criteria effectively. Recently, ? (?) proposed a supervised subset selection method using DPPs. Despite its superior performance, this method uses an exhaustive search for MAP inference, which makes it inapplicable for producing real-time summaries.

Local search has been widely used for submodular maximization subject to various constraints. This includes the analysis of greedy and local search by ? (?) providing a $1/(p+1)$ approximation guarantee for monotone submodular maximization under $p$ matroid constraints. For non-monotone submodular maximization, the most recent results include a $(1+O(1/\sqrt{p}))p$-approximation subject to a $p$-system constraints (?), a $1/5-\varepsilon$ approximation under $d$ knapsack constraints (?), and a $(p+1)(2p+2d+1)/p$-approximation for maximizing a general submodular function subject to a $p$-system and $d$ knapsack constraints (?).

Streaming algorithms for submodular maximization have gained increasing attention for producing online summaries. For monotone submodular maximization, ? (?) proposed a single pass algorithm with a $1/2\!-\!\epsilon$ approximation guarantee under a cardinality constraint $k$, using $O(k\log k/\epsilon)$ memory. Later, ? (?) provided a $1/4p$ approximation guarantee for the same problem under the intersection of $p$ matroid constraints. However, the required memory increases polylogarithmically with the size of the data. Finally, ? (?) presented deterministic and randomized algorithms for maximizing monotone and non-monotone submodular functions subject to a broader range of constraints, namely a $p$-matchoid. For maximizing a monotone submodular function, their proposed method gives a $1/4p$ approximation using $O(k\log k/\epsilon^{2})$ memory ($k$ is the size of the largest feasible solution). For non-monotone functions, they provide a deterministic $1/(9p+1)$ approximation using the $1\!/(p\!+\!1)$ offline approximation of ? (?). Their randomized algorithm provides a $1/(4p+1/\tau_{p})$ approximation in expectation, where $\tau_{p}=(1-\varepsilon)(2-o(1))/(ep)$ (?) is the offline approximation for maximizing a non-negative submodular function.

Using the monotone streaming algorithm of ? (?) with $1/4p$ approximation guarantee, our framework provides a $1/(4p+4\sqrt{p}+1)$ approximation for maximizing a non-monotone function under a $p$-matchoid constraint, which is a significant improvement over the work of ? (?). Note that any monotone streaming algorithm with approximation guarantee under a set of independence systems ${\cal I}$ (including a $p$-system constraint, once such an algorithm exists) can be integrated into our framework to provide approximations for non-monotone submodular maximization under the same set of independence systems ${\cal I}$, and $d$ knapsack constraints.

Problem Statement

We consider the problem of summarizing a stream of data by selecting, on the fly, a subset that maximizes a utility function $f:2^{V}\rightarrow\mathbb{R}_{+}$. The utility function is defined on $2^{V}$ (all subsets of the entire stream $V$), and for each $S\subseteq V$, $f(S)$ quantifies how well $S$ represents the ground set $V$. We assume that $f$ is submodular, a property that holds for many widely used such utility functions. This means that for any two sets $S\subseteq T\subseteq V$ and any element $e\in V\setminus T$ we have

[TABLE]

We denote the marginal gain of adding an element $e\in V$ to a summary $S\subset V$ by $f_{S}(e)=f(S\cup\{e\})-f(S)$. The function $f$ is monotone if $f_{S}(e)\geq 0$ for all $S\subseteq V$. Here, we allow $f$ to be non-monotone. Many data summarization applications can be cast as an instance of constrained submodular maximization under a set $\zeta\subseteq 2^{V}$ of constraints:

[TABLE]

In this work, we consider a collection of independence systems and multiple knapsack constraints. An independence system is a pair ${\cal M}^{I}=(V,\mathcal{I})$ where $V$ is a finite (ground) set, and $\mathcal{I}\subseteq 2^{V}$ is a family of independent subsets of $V$ satisfying the following two properties. (i) $\emptyset\in\mathcal{I}$, and (ii) for any $A\!\subseteq\!B\!\subseteq\!V$, ​$B\!\in\!\mathcal{I}$ implies that $A\in\mathcal{I}$ (hereditary property). A matroid $\mathcal{M}=(V,\mathcal{I})$ is an independence system with exchange property: if $A,B\in\mathcal{I}$ and $|B|>|A|$, there is an element $e\in B\setminus A$ such that $A\cup\{e\}\in\mathcal{I}$. The maximal independent sets of $\mathcal{M}$ share a common cardinality, called the rank of $\mathcal{M}$. A uniform matroid is the family of all subsets of size at most $l$. In a partition matroid, we have a collection of disjoint sets $B_{i}$ and integers $0\!\leq l_{i}\!\leq|B_{i}|$ where a set $A$ is independent if for every index $i$, we have $|A\cap B_{i}|\leq l_{i}.$ A $p$-matchoid generalizes matchings and intersection of matroids. For $q$ matroids ${\cal M}_{\ell}=(V_{\ell},\mathcal{I}_{\ell})$, $\ell\in[q]$, defined over overlapping ground sets $V_{\ell}$, and for $V=\cup_{\ell=1}^{q}V_{\ell}$, $\mathcal{I}=\{S\subseteq V:S\cap V_{\ell}\in\mathcal{I}_{\ell}~{}~{}\forall\ell\}$, we have that ${\cal M}^{p}=(V,\mathcal{I})$ is a $p$-matchoid if every element $e\in V$ is a member of $V_{\ell}$ for at most $p$ indices. Finally, a $p$-system is the most general type of constraint we consider in this paper. It requires that if $A,B\in{\cal I}$ are two maximal sets, then $|A|\leq p|B|$. A knapsack constraint is defined by a cost function $c:V\rightarrow\mathbb{R}_{+}$. A set $S\subseteq V$ is said to satisfy the knapsack constraint if $c(S)=\sum_{e\in S}c(e)\leq W$. Without loss of generality, we assume $W=1$ throughout the paper.

The goal in this paper is to maximize a (non-monotone) submodular function $f$ subject to a set of constraints $\zeta$ defined by the intersection of a collection of independence systems ${\cal I}$, and $d$ knapsacks. In other words, we would like to find a set $S\in{\cal I}$ that maximizes $f$ where for each set of knapsack costs $c_{i},i\in[d]$, we have $\sum_{e\in S}c_{i}(e)\leq 1$. We assume that the ground set $V=\{e_{1},\cdots,e_{n}\}$ is received from the stream in some arbitrary order. At each point $t$ in time, the algorithm may maintain a memory $M_{t}\!\subset\!V$ of points, and must be ready to output a feasible solution $S_{t}\subseteq M_{t}$, such that $S_{t}\in\zeta$.

Video Summarization with DPPs

Suppose that we are receiving a stream of video frames, e.g. from a surveillance or a wearable camera, and we wish to select a subset of frames that concisely represents all the diversity contained in the video. Determinantal Point Processes (DPPs) are good tools for modeling diversity in such applications. DPPs (?) are distributions over subsets with a preference for diversity. Formally, a DPP $\mathcal{P}$ on a set of items $V=\{1,2,...,N\}$ defines a discrete probability distribution on $2^{V}$​​, such that the probability of every $S\!\subseteq\!V$ is

[TABLE]

where $L$ is a positive semidefinite kernel matrix, and $L_{S}\equiv[L_{ij}]_{i,j\in S}$, is the restriction of $L$ to the entries indexed by elements of $S$, and $I$ is the $N\times N$ identity matrix. In order to find the most diverse and informative feasible subset, we need to solve the NP-hard problem of finding $\arg\max_{S\in\mathcal{I}}\det(L_{S})$ (?), where $\mathcal{I}\subset 2^{V}$ is a given family of feasible solutions. However, the logarithm $f(S)=\log\det(L_{S})$ is a (non-monotone) submodular function (?), and we can apply submodular maximization techniques.

Various constraints can be imposed while maximizing the above non-monotone submodular function. In its simplest form, we can partition the video into $T$ segments, and define a diversity-reinforcing partition matroid to select at most $k$ frames from each segment. Alternatively, various content-based constraints can be applied, e.g., we can use object recognition to select at most $k_{i}\geq 0$ frames showing person $i$, or to find a summary that is focused on a particular person or object. Finally, each frame can be associated with multiple costs, based on qualitative factors such as resolution, contrast, luminance, or the probability that the given frame contains an object. Multiple knapsack constraints, one for each quality factor, can then limit the total costs of the elements of the solution and enable us to produce a summary closer to human-created summaries by filtering uninformative frames.

Streaming algorithm for constrained submodular maximization

In this section, we describe our streaming algorithm for maximizing a non-monotone submodular function subject to the intersection of a collection of independence systems and $d$ knapsack constraints. Our approach builds on local search, a widely used technique for maximizing non-monotone submodular functions. It starts from a candidate solution $S$ and iteratively increases the value of the solution by either including a new element in $S\!$ or discarding one of the elements of $S$ (?). ? (?) showed that similar results can be obtained with much lower complexity by using algorithms for monotone submodular maximization, which, however, are run multiple times. Despite their effectiveness, these algorithms need multiple passes over the input and do not directly extend to the streaming setting, where we are only allowed to make one pass over the data. In the sequel, we show how local search can be implemented in a single pass in the streaming setting.

Streaming Local Search for a collection of independence systems

The simple yet crucial observation underlying the approach of ? (?) is the following. The solution obtained by approximation algorithms for monotone submodular functions often satisfy $f(S)\geq\alpha f(S\cup C^{*})$, where $1\geq\alpha>0$, and $C^{*}$ is the optimal solution. In the monotone case $f(S\cup C^{*})\geq f(C^{*})$, and we obtain the desired approximation factor $f(S)\geq\alpha f(C^{*})$. However, this does not hold for non-monotone functions. But, if $f(S\cap C^{*})$ provides a good fraction of the optimal solution, then we can find a near-optimal solution for non-monotone functions even from the result of an algorithm for monotone functions, by pruning elements in $S$ using unconstrained maximization. This still retains a feasible set, since the constraints are downward closed. Otherwise, if $f(S\cap C^{*})\leq\varepsilon\textsc{{OPT}}$, then running another round of the algorithm on the remainder of the ground set will lead to a good solution.

Backed by the above intuition, we aim to build multiple disjoint solutions simultaneously within a single pass over the data. Let IndStream be a single pass streaming algorithm for monotone submodular maximization under a collection of independence systems, with approximation factor $\alpha$. Upon receiving a new element from the stream, IndStream can choose (1) to insert it into its memory, (2) to replace one or a subset of elements in the memory by it, or otherwise (3) the element gets discarded forever. The key insight for our approach is that it is possible to build other solutions from the elements discarded by IndStream. Consider a chain of $q\!=\!\lceil 1/\sqrt{\alpha}\!+\!1\rceil$ instances of our streaming algorithm, i.e. $\{\textsc{{IndStream}}_{1},\cdots,\textsc{{IndStream}}_{q}\}$. Any element $e$ received from the stream is first passed to $\textsc{{IndStream}}_{1}$. If $\textsc{{IndStream}}_{1}$ discards $e$, or adds $e$ to its solution and instead discards a set $D_{1}$ of elements from its memory, then we pass the set $D_{1}$ of discarded elements on to be processed by $\textsc{{IndStream}}_{2}$. Similarly, if a set of elements $D_{2}$ is discarded by $\textsc{{IndStream}}_{2}$, we pass it to $\textsc{{IndStream}}_{3}$, and so on. The elements discarded by the last instance $\textsc{{IndStream}}_{q}$ are discarded forever. At any point in time that we want to return the final solution, we run unconstrained submodular maximization (e.g. the algorithm of ? (?)) on each solution $S_{i}$ obtained by $\textsc{{IndStream}}_{i}$ to get $S^{\prime}_{i}$, and return the best solution among $\{S_{i},S^{\prime}_{i}\}$ for $i\in[1,q]$.

Theorem 1**.**

Let IndStream be a streaming algorithm for monotone submodular maximization under a collection of independence systems ${\cal I}$ with approximation guarantee $\alpha$. Alg. 1 returns a set $S\in{\cal I}$ with

[TABLE]

using memory $O(M/\sqrt{\alpha})$, and average update time $O(T/\sqrt{\alpha})$ per element, where $M$ and $T$ are the memory and update time of IndStream.

The proof of all the theorems can be found in (?).

We make Theorem 1 concrete via an example: ? (?) proposed a $1/4p$-approximation streaming algorithm for monotone submodular maximization under a $p$-matchoid constraint. Using this algorithm as IndStream in Streaming Local Search, we obtain:

Corollary 2**.**

With Streaming Greedy of ? (?) as IndStream, Streaming Local Search yields a solution $S\in{\cal I}$ with approximation guarantee ${1}/(1+{2\sqrt{p}})^{2}$, using $O(\sqrt{p}k\log(k)/\varepsilon)$ memory and $O(p\sqrt{p}k\log(k)/\varepsilon)$ average update time per element, where $\mathcal{I}$ are the independent sets of a $p$-matchoid, and $k$ is the size of the largest feasible solution.

Note that any monotone streaming algorithm with approximation guarantee $\alpha$ under a collection of independence systems ${\cal I}$ can be integrated into Alg. 1 to provide approximation guarantees for non-monotone submodular maximization under the same set ${\cal I}$ of constraints. For example, as soon as there is a subroutine for monotone streaming submodular maximization under a $p$-system in the literature, one can use it in Alg. 1 as IndStream, and get the guarantee provided in Theorem 1 for maximizing a non-monotone submodular function under a $p$-system, in the streaming setting.

Streaming Local Search for independence systems and multiple knapsack constraints

To respect multiple knapsack constraints in addition to the collection of independence systems ${\cal I}$, we integrate the idea of a density threshold (?) into our local search algorithm. We use a (fixed) density threshold $\rho$ to restrict the IndStream algorithm to only pick elements if the function value per unit size of the selected elements is above the given threshold. We call this new algorithm IndStreamDensity. The threshold should be carefully chosen to be below the value/size ratio of the optimal solution. To do so, we need to know (a good approximation to) the value of the optimal solution OPT. To obtain a rough estimate of OPT, it suffices to know the maximum value $m=\max_{e\in V}f({e})$ of any singleton element: submodularity implies that $m\!\leq\!\textsc{{OPT}}\!\leq\!km$, where $k$ is an upper bound on the cardinality of the largest feasible solution satisfying all constraints. We update the value of the maximum singleton element on the fly (?), and lazily instantiate the thresholds to $\log(k)/\epsilon$ different possible values $(1+\epsilon)^{i}\in[\gamma,\gamma k]$, for $\gamma$ defined in Alg. 2. We show that for at least one of the discretized density thresholds we obtain a good enough solution.

Theorem 3**.**

Streaming Local Search* (outlined in Alg. 2) guarantees*

[TABLE]

with memory $O(M\log(k)/(\epsilon\sqrt{\alpha}))$, and average update time $O(T\log(k)/(\epsilon\sqrt{\alpha}))$ per element, where $k$ is an upper bound on the size of the largest feasible solution, and $M$ and $T$ are the memory and update time of the IndStream algorithm.

Corollary 4**.**

By using Streaming Greedy of ? (?), we get that Streaming Local Search has an approximation ratio $(1+\varepsilon)(1+4p+4\sqrt{p}+d(2+1/\sqrt{p}))$ with $O(\sqrt{p}k\log^{2}(k)/\varepsilon^{2})$ memory and update time $O(p\sqrt{p}k\log^{2}(k)/\epsilon^{2})$ per element, where $\mathcal{I}$ are the independent sets of the $p$-matchoid constraint, and $k$ is the size of the largest feasible solution.

Beyond the Black-Box.

Although the DPP probability in Eq. 1 only depends on the selected subset $S$, in many applications $f(S)$ may depend on the entire data set $V$. So far, we have adopted the common assumption that $f$ is given in terms of a value oracle (a black box) that computes $f(S)$. Although in practical settings this assumption might be violated, many objective functions are additively decomposable over the ground set $V$ (?). That means, $f(S)=\frac{1}{V}\sum_{e\in V}f_{e}(S)$, where $f_{e}(S)$ is a non-negative submodular function associated with every data point $e\in V$, and $f_{e}(.)$ can be evaluated without access to the full set $V$. For decomposable functions, we can approximate $f(S)$ by $f_{W}(S)=\frac{1}{W}\sum_{e\in W}f_{e}(S)$, where $W$ is a uniform sample from the stream (e.g. using reservoir sampling (?)).

Theorem 5** (? (?)).**

Assume that $f$ is decomposable, all of $f_{e}(S)$ are bounded, and w.l.o.g. $|f_{e}(S)|\!\leq\!1$. Let $W$ be uniformly sampled from $V$​. Then for $|W|\geq\frac{2k^{2}\log(2/\delta)+2k^{3}\log(V)}{\varepsilon^{2}}$, we can ensure that with probability $1\!-\!\delta$, Streaming Local Search guarantees

[TABLE]

Experiments

In this section, we apply Streaming Local Search to video summarization in the streaming setting. The main goal of this section is to validate our theoretical results and demonstrate the effectiveness of our method in practical scenarios, where the existing streaming algorithms are incapable of providing any quality guarantee for the solutions. In particular, for streaming non-monotone submodular maximization under a collection of independence systems and multiple knapsack constraints, none of the previous works provide any theoretical guarantees. We use the streaming algorithm of ? (?) for monotone submodular maximization under a $p$-matchoid constraint as IndStream, and compare the performance of our method111Our code is available at github.com/baharanm/non-mon-stream with exhaustive search (?), and a centralized method for maximizing a non-monotone submodular function under a $p$-system and multiple knapsack constraints, Fantom (?).

Dataset.

For our experiments, we use the Open Video Project (OVP), and the YouTube datasets with 50 and 39 videos, respectively (?). We use the pruned video frames as described in (?), where one frame is uniformly sampled per second, and uninformative frames are removed. Each video frame is then associated with a feature vector that consists of Fisher vectors (?) computed from SIFT features (?), contextual features, and features computed from the frame saliency map (?). The size of the feature vectors, $v_{i}$, are 861 and 1581 for the OVP and YouTube datasets.

The DPP kernel $L$ (Eq. 1), can be parametrized and learned via maximum likelihood estimation (?). For parametrization, we follow (?), and use both a linear transformation, i.e. $L_{ij}=v_{i}^{T}W^{T}Wv_{j}$, as well as a non-linear transformation using a one-hidden-layer neural network, i.e. $L_{ij}=z^{T}_{i}W^{T}Wz_{j}$ where $z_{i}=\tanh(Uv_{i})$, and $\tanh(.)$ stands for the hyperbolic transfer function. The parameters, $U$ and $W$ or just $W$, are learned on 80% of the videos, selected uniformly at random. By the construction of (?), we have $\det(L)>0$. However, $\det(L)$ can take values less than 1, and the function is non-monotone. We added a positive constant to the function values to make them non-negative. Following ? (?) for evaluation, we treat each of the 5 human-created summaries per video as ground truth for each video.

Sequential DPP.

To capture the sequential structure in video data, ? (?) proposed a sequential DPP. Here, a long video sequence is partitioned into $T$ disjoint yet consecutive short segments, and for selecting a subset $S_{t}$ from each segment $t\in[1,T]$, a DPP is imposed over the union of the frames in the segment $t$ and the selected subset $S_{t-1}$ in the immediate past frame $t-1$. The conditional distribution of the selected subset from segment $t$ is thus given by $\mathcal{P}(S_{t}|S_{t-1})=\frac{\det(L_{S_{t}\cup S_{t-1}})}{\det(I_{t}+L_{S_{t-1}\cup V_{t}})},$ where $V_{t}$ denotes all the video frames in segment $t$, and $I_{t}$ is a diagonal matrix in which the elements corresponding to $S_{t-1}$ are zeros and the elements corresponding to $S_{t}$ are 1. MAP inference for the sequential DPP is as hard as for the standard DPP, but submodular optimization techniques can be used to find approximate solutions. In our experiments, we use a sequential DPP as the utility function in all the algorithms.

Results.

Table 1 shows the F-score, Precision and Recall for our algorithm, that of ? (?) and Fantom (?), for segment size $|V_{t}|=10$. It can be seen that in all three metrics, the summaries generated by Streaming Local Search are competitive to the two centralized baselines.

Fig. 1a, 1g show the ratio of the F-score obtained by Streaming Local Search and Fantom vs. the F-score obtained by exhaustive search (?) for varying segment sizes, using linear embeddings on the YouTube and OVP datasets. It can be observed that our streaming method achieves the same solution quality as the centralized baselines. Fig. 1b, 1h show the speedup of Streaming Local Search and Fantom over the method of ? (?), for varying segment sizes. We note that both Fantom and Streaming Local Search obtain a speedup that is exponential in the segment size. In summary, Streaming Local Search achieves solution qualities comparable to (?), but 1700 times faster than (?), and 2 times faster than Fantom for larger segment size. This makes our streaming method an appealing solution for extracting real-time summaries. In real-world scenarios, video frames are typically generated at such a fast pace that larger segments make sense. Moreover, unlike the centralized baselines that need to first buffer an entire segment, and then produce summaries, our method generates real-time summaries after receiving each video frame. This capability is crucial in privacy-sensitive applications.

Fig. 1d and 1j show similar results for nonlinear representations, where a one-hidden-layer neural network is used to infer a hidden representation for each frame. We make two observations: First, non-linear representations generally improve the solution quality. Second, as before, our streaming algorithm achieves exponential speedup (Fig. 1e, 1k).

Finally, we also compared the three algorithms with a “standard”, non-sequential DPP as the utility function, for generating summaries of length 5% of the video length. Again, our method yields competitive performance with a much shorter running time (Fig. 1c, 1f, 1i, 1l).

Using constraints to generate customized summaries.

In our second experiment, we show how constraints can be applied to generate customized summaries. We apply Streaming Local Search to YouTube video 106, which is a part of America’s Got Talent series. It features a singer and three judges in the judging panel. Here, we generated two sets of summaries using different constraints. The top row in Fig. 2 shows a summary focused on the judges. Here we considered 3 uniform matroid constraints to limit the number of frames chosen containing each of the judges, i.e., ${\cal I}\!=\!\{S\!\subseteq\!V\!:|S\cap V_{j}|\leq l_{j}\}$, where $V_{j}\!\subseteq\!V$ is the subset of frames containing judge $j$, and $j\in[1,3]$; the $V_{j}$ can overlap. The limits for all the matroid constraints are $l_{j}=3$. To produce real-time summaries while receiving the video, we used the Viola-Jones algorithm (?) to detect faces in each frame, and trained a multiclass support vector machine using histograms of oriented gradients (HOG) to recognize different faces. The bottom row in Fig. 2 shows a summary focused on the singer using one matroid constraint.

To further enhance the quality of the summaries, we assigned different weights to the frames based on the probability for each frame to contain objects, using selective search (?). By assigning higher cost to the frames that have low probability of containing objects, and by limiting the total cost of the selected elements by a knapsack, we can filter uninformative and blurry frames, and produce a summary closer to human-created summaries. Fig. 3 compares the result of our method, the method of ? (?) and a human-created summary.

Conclusion

We have developed the first streaming algorithm, Streaming Local Search, for maximizing non-monotone submodular functions subject to a collection of independence systems and multiple knapsack constraints. In fact, our work provides a general framework for converting monotone streaming algorithms to non-monotone streaming algorithms for general constrained submodular maximization. We demonstrated its applicability to streaming video summarization with various personalization constraints. Our experimental results show that our method can speed up the summarization task more than 1700 times, while achieving a similar performance as centralized baselines. This makes it a promising approach for many real-time summarization tasks in machine learning and data mining. Indeed, our method applies to any summarization task with a non-monotone (nonnegative) submodular utility function, and a collection of independence systems and multiple knapsack constraints.

Acknowledgments.

This research was partially supported by ERC StG 307036, and NSF CAREER 1553284.

Supplementary Materials.

Analysis of Streaming Local Search

Proof of theorem 1

Proof.

Consider a chain of $r$ instances of our streaming algorithm, i.e. $\{\textsc{{IndStream}}_{1},\cdots,\textsc{{IndStream}}_{r}\}$. For each $i\in[1,r]$, $\textsc{{IndStream}}_{i}$ provides an $\alpha$-approximation guarantee on the ground set $V_{i}$ of items it has received. Therefore we have:

[TABLE]

where $C_{i}=C^{*}\cap V_{i}$ for all $i\in[1,r]$, and $C^{*}$ is the optimal solution. Moreover, for each $i$, $S^{\prime}_{i}$ is the solution of the unconstrained maximization algorithm on ground set $S_{i}$. Therefore, we have:

[TABLE]

where $\beta$ is the approximation guarantee of the unconstrained submodular maximization algorithm (Unconstrained-Max).

We now use the following lemma from (?) to bound the total value of the solutions provided by the $r$ instances of IndStream.

Lemma 6** (Lemma 2.2. of (?)).**

Let $f^{\prime}:2^{V}\rightarrow R$ be submodular. Denote by $A(p)$ a random subset of A where each element appears with probability at most $p$ (not necessarily independently). Then, $\mathbb{E}[f^{\prime}(A(p))]\geq(1-p)f^{\prime}(\emptyset)$.

Let $S$ be a random set which is equal to every one of the sets $\{S_{1},\cdots,S_{r}\}$ with probability $p=1/r$. For $f^{\prime}:2^{V}\rightarrow R$, and $f^{\prime}(S)=f(S\cup\textsc{{OPT}})$, from Lemma 6 we get:

[TABLE]

Also, note that each instance $i$ of IndStream in the chain has processed all the elements of the ground set $V$ except those that are in the solution of the previous instances of IndStream in the chain. As a result, $V_{i}=V\setminus\cup_{j=1}^{i-1}S_{i}$, and for every $i\in[1,r]$, we can write:

[TABLE]

Now, using Eq. 4, and via a similar argument as used in (?), we can write:

[TABLE]

Hence, we get:

[TABLE]

Taking the derivative w.r.t. $r$, we get that the ratio is maximized for $r=\left\lceil\sqrt{\frac{2\beta}{\alpha}}+1\right\rceil$. Plugging this value into Eq. 8, we have:

[TABLE]

Using $\beta=1/2$ from (?), we get the desired result:

[TABLE]

Finally, Corollary 2 follows by replacing $\alpha=1/4p$ from (?) and $\beta=1/2$ from (?):

[TABLE]

∎

For calculating the average update time, we consider the worst case scenario, where every element can go through the entire chain of $r$ instances of IndStream at some point during the run of Streaming Local Search. Here the total running time of the algorithm is $O(nrT)$, where $n$ is the size of the stream, and $T$ is the update time of IndStream. Hence the average update time per element for Streaming Local Search is $O(nrT/n)=O(rT)$.

Proof of theorem 3

Proof.

Here, a (fixed) density threshold $\rho$ is used to restrict the IndStream to only pick elements if $\frac{f_{S_{i}}(e)}{\sum_{j=1}^{d}c_{je}}\geq\rho$. We first bound the approximation guarantee of this new algorithm IndStreamDensity, and then use a similar argument as in the proof ot Theorem 1 to provide the guarantee for Streaming Local Search. Consider an optimal solution $C^{*}$ and set:

[TABLE]

By submodularity we know that $m\leq f(C^{*})\leq mk$, where $k$ is an upper bound on the cardinality of the largest feasible solution, and $m$ is the maximum value of any singleton element. Hence:

[TABLE]

Thus there is a run of the algorithm with density threshold $\rho\in R$ such that:

[TABLE]

For the run of the algorithm corresponding to $\rho$, we call the solution of the first instance $\textsc{{IndStreamDensity}}_{1}$, $S_{\rho}$. If $\textsc{{IndStreamDensity}}_{1}$ terminates by exceeding some knapsack capacity, we know that for one of the knapsacks $j\in[d]$, we have $c_{j}(S_{\rho})>1$, and hence also $\sum_{j=1}^{d}c_{j}(S_{\rho})>1$ (W.l.o.g. we assumed the knapsack capacities are 1). On the other hand, the extra density threshold we used for selecting the elements tells us that for any $e\in S_{\rho}$, we have $\frac{f_{S_{\rho}}(e)}{\sum_{j=1}^{d}c_{je}}\geq\rho$. I.e., the marginal gain of every element added to the solution $S_{\rho}$ was greater than or equal to $\rho\sum_{j=1}^{d}c_{je}$. Therefore, we get:

[TABLE]

Note that $S_{\rho}$ is not a feasible solution, as it exceeds the $j$-th knapsack capacity. However, the solution before adding the last element $e$ to $S_{\rho}$, i.e. $T_{\rho}=S_{\rho}-\{e\}$, and the last element itself are both feasible solutions, and by submodularity, the best of them provide us with the value of at least

[TABLE]

On the other hand, if $\textsc{{IndStreamDensity}}_{1}$ terminates without exceeding any knapsack capacity, we divide the elements in $C^{*}\setminus S_{\rho}$ into two sets. Let $C^{*}_{<\rho}$ be the set of elements from $C^{*}$ which cannot be added to $S_{\rho}$ because their density is below the threshold, i.e., $\frac{f_{S_{\rho}}(e)}{\sum_{i=1}^{d}c_{je}}<\rho$ and $C^{*}_{\geq\rho}$ be the set of elements from $C^{*}$ which cannot be added to $S_{\rho}$ due to independence system constraints. For the elements of the optimal solution $C^{*}$ which cannot be added to $S_{\rho}$ because their density is below the threshold, we have:

[TABLE]

Since $C_{<\rho}$ is a feasible solution, we know that $\sum_{e\in C_{<\rho}}c_{je}\leq 1$, and therefore:

[TABLE]

On the other hand, if the ground set was restricted to elements that pass the density threshold, then $S_{\rho}$ would be a subset of that ground set, and the approximation guarantee of $\textsc{{IndStream}}_{1}$ still holds; hence from Eq. 2 we know that:

[TABLE]

and thus we obtain:

[TABLE]

Adding Eq 11 and 12, and using submodularity we get:

[TABLE]

Therefore,

[TABLE]

Now, using a similar argument as in the proof of Theorem 1, we have:

[TABLE]

∎

Hence, we have:

[TABLE]

From Eq. 10, we know that $\rho\geq(1-\varepsilon)\rho^{*}$. Using Eq. 9, we get:

[TABLE]

Plugging in $r=\left\lceil\sqrt{\frac{2\beta}{\alpha}}+1\right\rceil$ and simplifying, we get the desired result:

[TABLE]

For $\beta=1/2$ from (?), we get the desired result:

[TABLE]

Corollary 4 follows by replacing $\alpha=1/4p$ from (?) and $\beta=1/2$ from (?):

[TABLE]

The average update time for one run of the algorithm corresponding to a $\rho\in R$ can be calculated as in the proof of Theorem 1. We run the algorithm for $\log(k)/\varepsilon$ different values of $\rho$, and hence the average update time of Streaming Local Search per element is $O(rT\log(k)/\varepsilon)$. However, the algorithm can be run in parallel for the $\log(k)/\varepsilon$ values of $\rho$ (line 7 of Algorithm 2), and hence using parallel processing, the average update time per element is $O(rT)$.