Unified Embedding and Metric Learning for Zero-Exemplar Event Detection

TL;DR

This paper introduces a unified embedding and metric learning approach for zero-exemplar event detection in videos, enabling effective retrieval of relevant videos based on textual descriptions without prior video examples.

Contribution

It proposes a joint embedding space for visual and textual data, improving zero-exemplar event detection by learning to measure event-video similarity end-to-end.

Findings

Outperforms state-of-the-art on TRECVID dataset

Effective in zero-exemplar event detection

Joint embedding improves retrieval accuracy

Abstract

Event detection in unconstrained videos is conceived as a content-based video retrieval with two modalities: textual and visual. Given a text describing a novel event, the goal is to rank related videos accordingly. This task is zero-exemplar, no video examples are given to the novel event. Related works train a bank of concept detectors on external data sources. These detectors predict confidence scores for test videos, which are ranked and retrieved accordingly. In contrast, we learn a joint space in which the visual and textual representations are embedded. The space casts a novel event as a probability of pre-defined events. Also, it learns to measure the distance between an event and its related videos. Our model is trained end-to-end on publicly available EventNet. When applied to TRECVID Multimedia Event Detection dataset, it outperforms the state-of-the-art by a considerable margin.

Tables

Table 1. Table 1: Comparison between the unified embedding and other baselines. The unified embedding model U achieves the best results on MED-13 and MED-14 datasets.

Baseline	Loss		Metric	$f_{𝒱}(\cdot )$	$f_{𝒯}(\cdot )$	MED13	MED14
modelV	${ℒ}_{mse}^{𝒱}$	(6)	✗	✓	✗	11.90	10.76
modelC	${ℒ}_{con}^{𝒞}$	(7)	✓	✓	✗	13.29	12.31
modelS	${ℒ}_{log}$	(4)	✗	✓	✓	15.60	13.49
modelN	${ℒ}_{mse}^{𝒩}$	(8)	✗	✓	✓	15.92	14.36
modelU	${ℒ}^{𝒰}$	(5)	✓	✓	✓	17.86	16.67

Table 2. Table 2: Performance comparison between our model and related works. We report the mean average precision (mAP%) for MED-13 and MED-14 datasets.

Method		MED13	MED14
TagBook [18]	ToM '15	12.90	05.90
Discovary [7]	ICAI '15	09.60	–
Composition [8]	AAAI '16	12.64	13.37
Classifiers [9]	CVPR '16	13.46	14.32
VideoStory† [17]	PAMI '16	15.90	05.20
VideoStory∗ [17]	PAMI '16	20.00	08.00
This Paper (modelU)		17.86	16.67

Table 3. Table 3: Our method improves over VideoStory when trained on the same dataset and using the same feature representation.

Method	Training Set	CNN Feat.	MED14
VideoStory	VideoStory46k [17]	GoogleNet	08.00
VideoStory	FCVID [53]	GoogleNet	11.84
VideoStory	EventNet [14]	GoogleNet	14.52
VideoStory	EventNet [14]	ResNet	15.80
This Paper	EventNet [14]	ResNet	16.67

Equations

$$t^z:v_i^z\succ v_j^z\to f(v_i^z,t^z)>f(v_j^z,t^z).$$

$${\mathcal{L}}_{con}$$

$$d_i$$

$${\mathcal{L}}_{log}=\sum _{i=1}^N\sum _{j=1}^M-y_i^j\log f_{\mathcal{T}}^j(k_i;W_{\mathcal{T}}).$$

$$\underset{W_{\mathcal{V}},W_{\mathcal{T}}}{\mathrm{argmin}}{\mathcal{L}}^{\mathcal{U}},$$

$${\mathcal{L}}^{\mathcal{U}}={\mathcal{L}}_{con}+{\mathcal{L}}_{log}.$$

$${\mathcal{L}}_{mse}^{\mathcal{V}}=\frac{1}{N}\sum _{i=1}^N\Vert y_i-f_{\mathcal{V}}(v_i;W_{\mathcal{V}}){\Vert }_2^2.$$

$${\mathcal{L}}_{con}^{\mathcal{C}}$$

$$d_i$$

$${\mathcal{L}}_{mse}^{\mathcal{N}}=\frac{1}{N}\sum _{i=1}^N\Vert f_{\mathcal{T}}(t_i;W_{\mathcal{T}})-f_{\mathcal{V}}(v_i;W_{\mathcal{V}}){\Vert }_2^2.$$

Full text

Related works train a bank of concept detectors on external data sources. These detectors predict confidence scores for test videos, which are ranked and retrieved accordingly. In contrast, we learn a joint space in which the visual and textual representations are embedded. The space casts a novel event as a probability of pre-defined events. Also, it learns to measure the distance between an event and its related videos.

Our model is trained end-to-end on publicly available EventNet. When applied to TRECVID Multimedia Event Detection dataset, it outperforms the state-of-the-art by a considerable margin.

1 Introduction

TRECVID Multimedia Event Detection (MED) [1, 2] is a retrieval task for event videos, with the reputation of being realistic. It comes in two flavors: few-exemplar and zero-exemplar, where the latter means that no video example is known to the model. Although expecting a few examples seems reasonable, in practice this implies that the user must already have an index of any possible query, making it very limited. In this paper, we focus on event video search with zero exemplars.

Retrieving videos of never-seen events, such as “renovating home”, without any video exemplar poses several challenges. One challenge is how to bridge the gap between the visual and the textual semantics [3, 4, 5]. One approach [3, 6, 7, 8, 9, 10] is to learn a dictionary of concept detectors on external data source. Then, scores for test videos are predicted using these detectors. Test videos are then ranked and retrieved accordingly. The inherent weakness of this approach is that the presentation of a test video is reduced to a limited vocabulary from the concept dictionary. Another challenge is how to overcome the domain difference between training and test events. While Semantic Query Generation (SQG) [3, 8, 9, 11] mitigates this challenge by extracting keywords from the event query, it does not address how relevant these keywords to the event itself. For example, keyword “person” is not relevant to event “car repair” as it is to “flash mob gathering”.

Our entry to zero-exemplar events is that they generally have strong semantic correlations [12, 13] with other possibly seen events. For instance, the novel event “renovating home” is related to “fit wall tiles”, “remove drywall”, or even to “paint door”. Novel events can, therefore, be casted on a repository of prior events, for which knowledge sources in various forms are available beforehand, such as the videos, as in EventNet [14], or articles, as in WikiHow [15]. Not only do these sources provide video examples of a large –but still limited– set of events, but also they provide an association of text description of events with their corresponding videos. A text article can describe the event in words: what is it about, what are the details and what are the semantics. We note that such a visual-textual repository of events may serve as a knowledge source, by which we can interpret novel event queries.

For Zero-exemplar Event Detection (ZED), we propose a neural model with the following novelties:

We formulate a unified embedding for multiple modalities (e.g. visual and textual) that enables a contrastive metric for maximum discrimination between events. 2. 2.

A textual embedding poses the representation of a novel event as a probability of predefined events, such that it spans a much larger space of admissible expressions. 3. 3.

We exploit a single data source, comprising pairs of event articles and related videos. A single source rather enables end-to-end learning from multi-modal individual pairs.

We empirically shows that our novelties result in performance improvement. We evaluate the model on TRECVID Multimedia Event Detection (MED) 2013 [1] and 2014 [2]. Our results show significant improvement over the state-of-the-art.

2 Related Work

We identify three families of methods for ZED, as in figure 2 (a), (b) and (c).

Visual Embedding and Textual Retrieval. As in figure 2(a), given a video $v_{i}$ represented as $x\in\mathcal{X}$ and a related text $t$ represented as $y\in\mathcal{Y}$. Then, a visual model $f_{\mathcal{V}}$ is trained to project $x$ as $y^{v}\in\mathcal{Y}$ such that the distance is minimized between $\left(y^{v},y\right)$. In test time, video ranking and retrieval is done using distance metric between the projected test video $y^{t}$ and test query representation $y$.

[16, 17] project the visual feature $x$ of a web video $v$ into term-vector representation $y$ of the video’s textual title $t$. However, during training, the model makes use of the text query of the test events to learn better term-vector representation. Consequently, this limits the generalization for novel event queries.

Textual Embedding and Visual Retrieval. As in figure 2(b), a given text query $t$ is projected into $x^{t}\in\mathcal{X}$ using pre-trained or learned language model $f_{\mathcal{T}}$.

[18] makes use of freely-available weekly-tagged web videos. Then it propagates tags to test videos from its nearest neighbors. Methods [7, 8, 9, 10, 3] have similar approach. Given a text query $t$, Semantic Query Generation (SQG) extracts $N$ most related concepts $\left\{c_{i},i\in N\right\}$ to the test query. Then, pre-trained concept detectors predict probability scores $\left\{s_{i},i\in N\right\}$ for a test video $v$. Aggregating these probabilities results in the final video score $s_{v}$, upon which videos are ranked and retrieved. [9] learns weighted averaging.

The shortcoming of this family is that expressing a video as probability scores of few concepts is under-representation. Any concept that exists in the video but is missing in the concept dictionary is thus unrepresented.

Visual-Textual Embedding and Semantic Retrieval. As in figure 2(c), visual $f_{\mathcal{V}}$ and textual $f_{\mathcal{T}}$ models are trained to project both of the visual $x$ and textual $y$ features into a semantic space $\mathcal{Z}$. During test, ranking score is the distance between the projections $z^{v},z^{t}$ in the semantic space $\mathcal{Z}$.

[19] projects video concepts into a high-dimensional lexicon space. Separately, it projects concept-based features to the space, which overcomes the lexicon mismatch between the query and the video concepts.  [20] embeds a fusion of low and mid-level visual features into distributional semantic manifold [21, 22]. In a separate step, it embeds text-based concepts into the manifold.

The third family, see figure 2(c), is superior to the others, see figure 2(a), (b). However, one drawback of [19, 20] is separately embedding both the visual and textual features $z^{v},z^{t}$. This leads to another drawback, having to measure the distance between $\left(z^{v},z^{t}\right)$ in a post-processing step (e.g. cosine similarity).

Unified Embedding and Metric Learning Retrieval Our method rather falls into a new family, see figure 2(d), and it overcomes the shortcomings of [19, 20] by the following. It is trained on a single data source, enabling a unified embedding for features of multiple modalities into a metric space. Consequently, the distance between the embedded features is measured by the model using the learned metric space.

Auxiliary Methods

Independent to the previous works, the following techniques have been used to improve the results: self-paced reranking [23], pseudo-relevance feedback [24], event query manual intervention [25], early fusion of features (action [26, 27, 28, 29, 30] or acoustic [31, 32, 33]) or late fusion of concept scores [17]. All these contributions may be applied to our method.

Visual Representation. ConvNets [34, 35, 36, 37] provide frame-level representation. To tame them into video-level counterpart, literature use: i- frame-level filtering [38] ii- vector encoding [39, 40] iii- learned pooling and recounting  [10, 41] iv- average pooling [16, 17]. Also, low-level action [28, 29], mid-level action [26, 27] or acoustic [31, 32, 33] features can be used. Textual Representation. To represent text, literature use: i- sequential models [42] ii- continuous word-space representations [22, 43] iii- topic models [44, 45] iv- dictionary-space representation [17].

3 Method

3.1 Overview

Our goal is zero-exemplar retrieval of event videos with respect to their relevance to a novel textual description of an event. More specifically, for the zero-exemplar video dataset $\mathcal{D}^{z}=\{v^{z}_{i}\},i=1,\dots,L$ and given any future, textual event description $t^{z}$, we want to learn a model $f(\cdot)$ that ranks the videos $v^{z}_{i}$ according to the relevance to $t^{z}$, namely:

[TABLE]

3.2 Model

Since we focus on zero-exemplar setting, we cannot expect any training data directly relevant to the test queries. As such, we cannot directly optimize our model for the parameters $W_{\mathcal{T}},W_{\mathcal{V}}$ in eq. (3). In the absence of any direct data, we resort to external knowledge databases. More specifically, we propose to cast future novel query descriptions as a convex combination of known query descriptions in external databases, where we can measure their relevance the database videos.

We start from a dataset $\mathcal{D}^{z}=\left\{v_{i},k_{i},l_{j},t_{j}\right\},\;i=1,\dots,N,\;j=1,\dots,M$ organized by an event taxonomy, where we do not neither expect nor require the events to overlap with any future event queries. The dataset is composed of $M$ events. Each event is associated with a textual, article description of the event, analyzing different aspects of it, such as: (i) the typical appearance of subjects and objects (ii) it’s procedures (iii) the steps towards completing task associated with it. The dataset contains in total $N$ videos, with $v_{i}$ denoting the $i$-th video in the dataset with metadata $k_{i}$, e.g. the title of the video. A video is associated with an event label $l_{i}$ and the article description $t_{i}$ of the event it belongs to. Since multiple videos belong to the same event, they share the article description of such event.

The ultimate goal of our model is zero-exemplar search for event videos. Namely, provided unknown text queries by the user, we want to retrieve those videos that are relevant. We illustrate our proposed model during training in figure 3. The model is composed of two components, a textual embedding $f_{\mathcal{T}}(\cdot)$, a visual embedding $f_{\mathcal{V}}(\cdot)$. Our ultimate goal is the ranking of videos, $v_{i}\succ v_{j}\succ v_{k}$ with respect to their relevance to a query description, or in pairwise terms $v_{i}\succ v_{j}$, $v_{j}\succ v_{k}$ and $v_{i}\succ v_{k}$.

Let us assume a pair of videos $v_{i}$, $v_{j}$ and query description $t$, where video $v_{i}$ is more relevant to the query $t$ than $v_{j}$. Our goal is a model that learns to put videos in the correct relative order, namely $(v_{i},t)\succ(v_{j},t)$. This is equivalent to a model that learns visual-textual embeddings such that $d^{tv}_{i}<d^{tv}_{j}$, where $d^{tv}_{i}$ is the distance between visual-textual embeddings of $(v_{i},t)$, $d^{tv}_{j}$ is the same for $(v_{j},t)$. Since we want to compare distances between pairs $(v_{i},t),(v_{j},t)$, we pose the learning of our model as the minimization of a contrastive loss [46]:

[TABLE]

where $f_{\mathcal{T}}(t_{i};W_{\mathcal{T}})$ is the projection of the query description $t_{i}$ into the unified metric space $\mathcal{Z}$ parameterized by $W_{\mathcal{T}}$, $f_{\mathcal{V}}(v_{i};W_{\mathcal{V}})$ is the projection of a video $v_{i}$ onto the same space $\mathcal{Z}$ parameterized by $W_{\mathcal{V}}$ and $h_{i}$ a target variable that equals to $1$ when the $i$-th video is relevant to the query description $t_{i}$ and [math] otherwise. Naturally, to optimize eq. (2), we first need to define the projections $f_{\mathcal{T}}(\cdot;W_{\mathcal{T}})$ and $f_{\mathcal{V}}(\cdot;W_{\mathcal{V}})$ in eq. (3).

Textual Embedding. The textual embedding component of our model, $f_{\mathcal{T}}(\cdot;W_{\mathcal{T}})$, is illustrated in figure 3 (top). This component is dedicated to learn a projection of a textual input –including any future event queries $t$– on to the unified space $\mathcal{Z}$. Before detailing our model $f_{\mathcal{T}}$, however, we note that that the textual embedding can be employed not only with event article descriptions, but also with any other textual information that might be associated to the dataset videos, such as textual metadata. Although we expect the video title not to be as descriptive as the associated article, they may still be able to offer some discriminative information as previously shown [16, 17] which can be associated to the event category.

We model the textual embedding as a shallow (two layers) multi-layer perceptron (MLP). For the first layer we employ a ReLU nonlinearity. The second layer serves a dual purpose. First, it projects the article description of an event on the unified space $\mathcal{Z}$. This projection is category-specific, namely different videos that belong to the same event will share the projection. Second, it can project any video-specific textual metadata into the unified space. We, therefore, propose to embed the title metadata $k_{i}$, which is uniquely associated with a video, not an event category. To this end, we opt for softmax nonlinearity for the second layer, followed by an additional logistic loss term to penalized misprediction of titles $m_{i}$ with respect to the video’s event label $y_{i}^{j}$, namely

[TABLE]

Overall, the textual embedding $f_{\mathcal{T}}^{j}$ is trained with a dual loss in mind. The first loss term, see eq. (2) (3) takes care that the final network learns event-relevant textual projections. The second loss term, see eq. (4), takes care that the final network does not overfit to the particular event article descriptions. The latter is crucial because the event article descriptions in $\mathcal{D}^{z}$ will not overlap with the future event queries, since we are in a zero-exemplar retrieval setting. As such, training the textual embedding to be optimal only for these event descriptions will likely result in severe overfitting. Our goal and hope is that the final textual embedding model $f_{\mathcal{T}}$ will capture both event-aware and video-discriminative textual features.

Visual Embedding. The visual embedding component of our model, $f_{\mathcal{V}}(\cdot;W_{\mathcal{V}})$, is illustrated in figure 3 (bottom). This component is dedicated to learn a projection from the visual input, namely the videos in our zero-exemplar dataset $\mathcal{D}^{z}$, into the unified metric space $\mathcal{Z}$. The goal is to project the videos belonging to semantically similar events; project them into a similar region in the space. We model the visual embedding $f_{\mathcal{V}}(v_{i};W_{\mathcal{V}})$ using a shallow (two layers) multi-layer perceptron with tanh nonlinearities, applied to any visual feature for video $v_{i}$.

End-to-End Training. At each training forward-pass, the model is given a triplet of data inputs, an event description $t_{i}$, a related video $v_{i}$ and video title $k_{i}$. From eq. (3) we observe that the visual embedding $f_{\mathcal{V}}(v_{i};W_{\mathcal{V}})$ is encouraged to minimize its distance with the output of the textual embedding $f_{\mathcal{T}}(t_{i};W_{\mathcal{T}})$. In the end, all the modules of the proposed model are differentiable. Therefore, we train our model in an end-to-end manner by minimizing the following objective

[TABLE]

For the triplet input $\left(v_{i},t_{i},k_{i}\right)$, we rely on external representations, since our ultimate goal is zero-exemplar search. Strictly speaking, a visual input $v_{i}$ is represented as CNN [35] feature vector, while textual inputs $t_{i},k_{i}$ are represented as LSI [45] or Doc2Vec [43] feature vectors. However, given that these external representations rely on neural network architectures, if needed, they could also be further fine-tuned. We choose to freeze CNN and Doc2Vec modules to speed up training. Finally, in this paper we refer to our main model with unified embedding, as modelU.

Inference. After training, we fix the parameters ($W_{\mathcal{V}},W_{\mathcal{T}})$. At test time, we set our function $f(\cdot)$ from eq. (1) to be equivalent to the distance function from eq. (LABEL:eq:distance). Hence, at test time, we compute the Euclidean distance in the learned metric space $\mathcal{Z}$ between the embeddings $\left(z^{v},z^{t}\right)$ of test video $v$ and novel event description $t$, respectively.

4 Experiments

4.1 Datasets

Before delving into the details of our experiments, first we describe the external knowledge sources we use.

Training dataset. We leverage videos and articles from publicly available datasets. EvenNet [14] is a dataset of $\sim$90k event videos, harvested from YouTube and categorized into 500 events in hierarchical form according to the events’ ontology. Each event category contains around $180$ videos. Each video is coupled with a text title, few tags and related event’s ontology.

We exploit the fact that all events in EventNet are harvested from WikiHow [15] – a website for How-To articles covering a wide spectrum of human activities. For instance: “How to Feed a Dog” or “How to Arrange Flowers”. Thus, we crawl WikiHow to get the articles related to all the events in EventNet.

Test dataset. As the task is zero-exemplar, the test sets are different from the training. While EventNet serves as the training, the following serve as the test: TRECVID MED-13 [1] and MED-14 [1]. In details, they are datasets of videos for events. They comprise 27k videos. There are two versions, MED-13 and MED-14 with 20 events for each. Since 10 events overlap, the result is 30 different events in total. Each event is coupled with short textual description (title and definition).

4.2 Implementation Details

Video Features. To represent a video $v$, we uniformly sample a frame every one second. Then, using ResNet [35], we extract pool5 CNN features for the sampled frames. Then, we average pool the frame-level features to get the video-level feature $x^{v}$. We experiment different features from different CNN models: ResNet (prob, fc1000), VGG [37] (fc6, fc7), GoogLeNet [47] (pool5, fc1024), and Places365 [48] (fc6, fc7,fc8) except we find ResNet pool5 to be the best. We only use ResNet pool5 and we don’t fuse multiple CNN features.

Text Features. We choose topic modeling [44, 45], as it is well-suited for long (and sometimes noisy) text articles. We train LSI topic model [45] on Wikipedia corpus [49]. We experiment different latent topics ranging from $300$ to $6000$, expect we found 2500 to be the best. Also, we experiment other textual representations as LDA [44], SkipThoughts [50] and Doc2Vec [43]. To extract a feature from an event article $k$ or video title $t$, first we preprocess the text using standard MLP steps: tokenization, lemmatization and stemming. Then, for $k,t$ we extract 2500-D LSI features $y^{k},y^{t}$, respectively. The same steps apply to MED text queries.

Model Details. Our visual and textual embeddings $f_{\mathcal{V}}(\cdot),f_{\mathcal{T}}(\cdot)$ are learned on top of the aforementioned visual and textual features ($x^{v},y^{k},y^{t}$). $f_{\mathcal{T}}(\cdot)$ is a 1-hidden layer MLP classifier with ReLU for hidden, softmax for output, logistic loss and 2500-2500-500 neurons for the input, hidden, and output layers, respectively. Similarly, $f_{\mathcal{V}}(\cdot)$ is a 1-hidden layer MLP regressor with ReLU for hidden, contrastive loss and 2048-2048-500 neurons for the input, hidden, and output layers, respectively. Our code is made public111github.com/noureldien/unified_embedding to support further research.

4.3 Textual Embedding

Here, we qualitatively demonstrate the benefit of the textual embedding $f_{\mathcal{T}}(\cdot)$. Figure 4 shows the similarity matrix between MED and EventNet events. Each dot represents how a MED event is similar to EventNet events. It shows that our embedding (right) is better than LSI (left) in mapping MED to EventNet events. For example, LSI wrongly maps “9: getting a vehicle unstuck” to “256: launch a boat” while our embedding correctly maps it to “170: drive a car”. Also, our embedding maps with higher confidence than LSI, as in “16: doing homework or study”.

Figure 5 shows the similarity matrix for MED events, where each dot represents how related any MED event to all the others. Our textual embedding (right) is more discriminant than on the LSI feature representation (left). For example, LSI representation shows high semantic correlation between events “34: fixing musical instrument” and “40: tuning musical instrument”, while our embedding discriminate them.

Next, we quantitatively demonstrate the benefit of the textual embedding $f_{\mathcal{T}}(\cdot)$. In contrast to the main model, see section 3, we investigate baseline modelV, where we discard the textual embedding $f_{\mathcal{T}}(\cdot)$ and consider only the visual embedding $f_{\mathcal{V}}(\cdot)$. We project a video $v$ on the LSI representation $y$ of the related event $t$. Thus, this baseline falls in the first family of methods, see figure 2(a). It is optimized using mean-squared error (MSE) loss $\mathcal{L}^{\mathcal{V}}_{mse}$, see eq. 6. The result of this baseline is reported in section 5, table 1.

[TABLE]

Also, we train another baseline modelC, which is similar to the aforementioned V except instead of using MSE loss $\mathcal{L}^{\mathcal{V}}_{mse}$, see eq. (6), it uses contrastive loss $\mathcal{L}^{\mathcal{C}}_{con}$, as follows:

[TABLE]

4.4 Unified Embedding and Metric Learning

In this experiment, we demonstrate the benefit of the unified embedding. In contrast to our model presented in section 3, we investigate baseline modelS, where this baseline does not learn joint embedding. Instead, it separately learns visual $f_{\mathcal{V}}(\cdot)$ and textual $f_{\mathcal{T}}(\cdot)$ projections. We model these projections as a shallow (2-layer) MLP trained to classify the data input into 500 event categories, using logistic loss, same as eq. (4).

We conduct another experiment to demonstrate the benefit of learning metric space. In contrast to our model presented in section 3, we investigate baseline modelN, where we discard the metric learning layer. Consequently, this baseline learns the visual embedding is a shallow (2 layers) multi-layer perceptron with tanh non linearities. Also, we replace the contrastive loss $\mathcal{L}_{c}$, see eq. (2) with mean-squared error loss $\mathcal{L}_{mse}$, namely

[TABLE]

During retrieval, this baseline embeds a test video $v_{i}$ and novel text query $t_{i}$ as features $z^{v},z^{t}$ onto the common space $\mathcal{Z}$ using textual and visual embeddings $f_{\mathcal{T}}(\cdot),f_{\mathcal{V}}(\cdot)$, respectively. However, in a post-processing step, retrieval score $s_{i}$ for the video $v_{i}$ is the cosine distance between $\left(z^{v},z^{t}\right)$. Similarly, all test videos are scored, ranked and retrieved. The results of the aforementioned baselines modelS and modelN are reported in table 1.

Comparing Different Embeddings. In the previous experiments, we investigated several baselines of the unified embedding (modelU), namely visual-only embedding (modelV), separate visual-textual embedding (modelS) and non-metric visual-textual embedding (modelN). In a qualitative manner, we compare the results of such embeddings. As shown in figure 6, we use these baselines to embed event videos of MED-13 and MED-14 datasets into the corresponding spaces. At the same time, we project the textual description of the events on the same space. Then, we use t-SNE [51] to visualize the result on 2D manifold. As seen, the unified embedding, see sub-figures 6(e), 6(f) learns more discriminant representations than the other baselines, see sub-figures 6(a), 6(b), 6(c) and  6(d). The same observation holds for both for MED-13 and MED-14 datasets.

4.5 Mitigating Noise in EventNet

Based of quantitative and qualitative analysis, we conclude that EventNet is noisy. Not only videos are unconstrained, but also some of the video samples are irrelevant to their event categories. EvenNet dataset [14] is accompanied by 500-category CNN classifier. It achieves top-1 and top-5 accuracies of 30.67% and 53.27%, respectively. Since events in EventNet are structured as an ontological hierarchy, there is a total of 19 high-level categories. The classifier achieves top-1 and top-5 accuracies of 38.91% and 57.67%, respectively, over these high-level categories.

Based on these observations, we prune EventNet to remove noisy videos. To this end, first we represent each video as average pooling of ResNet pool5 features. Then, we follow the conventional 5-fold cross validation with 5 rounds. For each round, we split the dataset into 5 subsets, 4 subsets $\mathcal{V}_{t}$ for training and the last $\mathcal{V}_{p}$ for pruning. Then we train a 2-layer MLP for classification. After training, we forward-pass the videos of $\mathcal{V}_{p}$ and rule-out the mis-classified ones.

The intuition behind pruning is that we rather learn salient event concepts using less video samples than learn noisy concepts with more samples. Pruning reduced the total number of videos by 26%, from 90.2k to 66.7k. This pruned dataset is all what we use in our experiments.

4.6 Latent Topics in LSI

When training LSI topic model on Wikipedia corpus, a crucial parameter is the number of latent topics $K$ the model constructs. We observe improvements in the performance directly proportional to increasing $K$. The main reason that the bigger the value of $K$, the more discriminant the LSI feature is. Figure 7 confirms our understanding.

5 Results

Evaluation metric. Since we are addressing, in essence, an information retrieval task, we rely on the average precision (AP) per event, and mean average precision (mAP) per dataset. We follow the standard evaluation method as in the relevant literature  [1, 2, 52].

Comparing against model baselines. In table 1, we report the mAP score of our model baselines, previously discussed in the experiments, see section 4. The table clearly shows the marginal contribution of each of novelty for the proposed method.

Comparing against related work. We report the performance of our method, the unified embedding modelU on TRECVID MED-13 and MED-14 datasets. When compared with the related works, our method improves over the state-of-the-art by a considerable margin, as shown in table 2 and figure 8.

It is important to point out that VideoStory† uses only object feature representation, so its comparable to our method. However, VideoStory∗ uses motion feature representation and expert text query (i.e. using term-importance matrix $\bm{H}$ in  [17]). To rule out the marginal effect of using different datasets and features, we train VideoStory and report results in table 3. Clearly, CNN features and video exemplars in the training set can improve the model accuracy, but our method improves against VideoStory when trained on the same dataset and using the same features. Other works (Classifiers [9], Composition [8]) use both image and action concept classifiers. Nonetheless, our method improves over them using only object-centric CNN feature representations.

6 Conclusion

In this paper, we presented a novel approach for detecting events in unconstrained web videos, in a zero-exemplar fashion. Rather than learning separate embeddings form cross-modal datasets, we proposed a unified embedding where several cross-modalities are jointly projected. This enables end-to-end learning. On top of this, we exploited the fact that zero-exemplar is posed as retrieval task and proposed to learn metric space. This enables measuring the similarities between the embedded modalities using this very space.

We experimented the novelties and demonstrated how they contribute to improving the performance. We complemented this by improvements over the state-of-the-art by considerable margin on MED-13 and MED-14 datasets.

However, the question still remains, how can we discriminate between these two MED events “34: fixing musical instrument” and “40: tuning musical instrument”. We would like to argue that temporal modeling for human actions in videos is of absolute necessity to achieve such fine-grained event recognition. In future research, we would like to focus on human-object interaction in videos and how to model it temporally.

Acknowledgment

We thank Dennis Koelma, Masoud Mazloom and Cees Snoek222{kolema,m.mazloom,cgmsnoek}@uva.nl for lending their insights and technical support for this work.