Less is More - diveXplore 5.0 at VBS 2021

TL;DR

diveXplore 5.0 is a redesigned, modular video exploration system that simplifies complexity to improve performance in interactive video search challenges.

Contribution

the paper presents a complete rebuild of diveXplore 5.0 focusing on modularity and reduced complexity to enhance system performance.

Findings

performance improved in VBS 2021

system complexity significantly reduced

maintained useful features in a modular design

Abstract

As a longstanding participating system in the annual Video Browser Showdown (VBS2017-VBS2020) as well as in two iterations of the more recently established Lifelog Search Challenge (LSC2018-LSC2019), diveXplore is developed as a feature-rich Deep Interactive Video Exploration system. After its initial successful employment as a competitive tool at the challenges, its performance, however, declined as new features were introduced increasing its overall complexity. We mainly attribute this to the fact that many additions to the system needed to revolve around the system's core element - an interactive self-organizing browseable featuremap, which, as an integral component did not accommodate the addition of new features well. Therefore, counteracting said performance decline, the VBS 2021 version constitutes a completely rebuilt version 5.0, implemented from scratch with the aim of greatly reducing the system's complexity as well as keeping proven useful features in a modular manner.

Full text

1 Introduction

The annual video retrieval challenge known as the Video Browser Showdown [12, 7, 8, 10] (VBS) as well as the newly developed Lifelog Search Challenge [3, 4] (LSC) based on personal lifelog data are team competitions that involve quick interactive search of large multimodal databases. As such they include multiple tasks, such as Known-Item Search (KIS) along with Ad-hoc Video Search111https://www-nlpir.nist.gov/projects/tv2018/Tasks/ad-hoc/ (AVS) as well as two competition modes featuring experts and novices using the various competing systems. These conditions, coupled with a relatively tight per-task time-limit of merely a few minutes, in general imposes two requirements on a participating system: the need to be as sophisticated as possible in order to retrieve meaningful results, while at the same time being as simple to use as possible for accommodating novice users familiar with neither the tasks nor the systems.

The diveXplore system [14, 9, 5, 13, 6] continuously participated in the VBS challenge since its 5th iteration (VBS2017-VBS2020) and grew in complexity along with the size of the competition’s gradually growing dataset [11], which currently comprises approximately 1000 hours of video [2]. Initially developed as a video exploration tool utilizing the concept of self-organizing feature maps [1], many new and exploratory features were added to the system for each VBS iteration, which led to an increasing deviation from its original core feature: video exploration through browsing. Consequently, the system became hard to maintain and complex to alter. Therefore, in order to competitively participate in VBS2021, diveXplore 5.0 is rebuilt from scratch focusing on the integration of successful features from the past challenges, while still offering video exploration features at a more modular level, preventing unnecessary feature dependencies. Figure 1 portrays the system’s new architecture, technology and features. The remaining sections describe diveXplore in detail as well as highlight novel additions to the system.

2 diveXplore 5.0

As a veteran system at the VBS, diveXplore was first developed for VBS2017 [14] incorporating an approach similar to the winning system at VBS2016 [1] for its main interface: an interactive, collaborative and self-organizing featuremap. Users were able to organize this map according to several similarity features, such as deep concepts, color, texture and motion. Additionally, these features could be utilized for similarity search on a keyframe basis with results being shown in separate views. Over the course of the last VBS iterations, these views became more and more important, as new features were introduced, such as a shot-aware video player, storyboards, color filter and textual concept as well as hand-drawn sketch search. All of these components needed to be integrated in accordance with the ever-present featuremap, such that navigating shots or videos in any of the views required re-positioning the currently displayed featuremap or even loading a new one. This overall slowed system performance in addition to being very error-prone. Therefore, the focus of diveXplore 5.0 is to remove these strong dependencies and construct separate views that do not have to be updated during most interactions.

2.1 Architecture

As illustrated in Figure 1, the system builds on several offline preprocessing steps in order to facilitate smooth interactive retrieval during the competition. First, the V3C1 dataset is split into custom shots as well as uniformly sampled frames using a one-second interval. This makes video retrieval very flexible since query results can be ordered on different levels of granularity. Subsequently, all included analyses can be started, which mostly include feature extraction utilizing hand-crafted as well as deep neural network approaches. The technologies used for creating the systems are web-based. The back end comprises two NodeJs222https://nodejs.org servers, where one is used for hosting keyframes, thumbnails and featuremaps, while the other server acts as a RESTful middleware between a client written in Angular 333https://angular.io and a MongoDB 444https://www.mongodb.com database.

2.2 Features

As mentioned, diveXplore 5.0 keeps the most useful features from previous VBS iterations, while adjusting them to accommodate aforementioned analyses on shot as well as frame level. Consequently, it still provides a textual concept search that allows for the search of different deep concepts such as objects, attributes, locations as well as metadata. Furthermore, the possibility for similarity search is maintained as well, especially since it proved invaluable for AVS tasks: a user can select a particular shot or frame and can retrieve similar items by choosing a similarity measure such as deep concepts, color, texture or motion features. In addition, the system includes two additional explorative views that borrow from previously integrated components, yet incorporate novel strategies. The following sections describe these new features in more detail – concept-based featuremaps and a video-based similarity filter.

2.2.1 Concept-based Featuremaps

Figure 1 already indicates that diveXplore 5.0 keeps the concept of featuremaps, albeit in a different form: the large map incorporating the shots of all videos organized by different concepts is removed while maintaining and organizing smaller maps for individual concepts such as objects, which is illustrated in Figure 2. By entering valid concepts, such as ’car’ in the example, users can retrieve one or more featuremaps depending on how many incorporated deep nets include this particular concept. These smaller maps can subsequently be organized by utilizing all measures available for similarity search as well as recognition confidence and video affiliation. This feature is available for the multi-map overview shown in Figure 2 as well as an individual fullscreen view after selecting an individual featuremap.

2.2.2 Video-based Similarity Filter

The video-based similarity filter, shown in Figure 3, is a new addition based on the system’s previously integrated storyboard feature, which provided a user with small summaries of each individual video. The main difference to the former implementation is the addition of interaction through several combinable filters. The view as well makes use of deep concepts and similarity, but for comparing entire videos or sections of them. As an example, the figure shows a situation where full videos are ordered by their creation period, i.e. between 2008 and 2012. Similarly, a user can as well select one or multiple concepts such as ’person’ and ’apple’ and group video sections according to either the frequency or classification confidence of these terms. Since this view allows for a multitude of different combinations it will expectedly be subject to many improvements in future system updates.

3 Conclusion

We present diveXplore 5.0 – a completely rebuilt version of a long-time competing system at the annual Video Browser Showdown. While keeping some proven useful features, such as concept and similarity search, the system includes two additional views for dataset exploration: concept-based featuremaps enabling the exploration and organization of individual concepts, as well as a video-based similarity filter, which allows for video- or segment-wide similarity search according to various combinable criteria. All features are integrated on a shot- as well as uniformly sampled frame-basis in order to increase the system’s flexibility. We expect these compared to diveXplore 4.0 much reduced and more carefully integrated components to shape a competitive system for VBS2021.

Acknowledgments

This work was funded by the FWF Austrian Science Fund under grant P 32010-N38.