VStore: A Data Store for Analytics on Large Videos

TL;DR

VStore is a novel data store that optimizes large video analytics by automatically configuring video formats through backward derivation, enabling fast, resource-efficient processing of large archival videos.

Contribution

It introduces backward derivation of configuration to automatically optimize video formats for multiple resources in large-scale video analytics.

Findings

Supports over 100 configuration knobs.

Achieves up to 362x real-time video query speed.

Automatically derives complex video format configurations.

Abstract

We present VStore, a data store for supporting fast, resource-efficient analytics over large archival videos. VStore manages video ingestion, storage, retrieval, and consumption. It controls video formats along the video data path. It is challenged by i) the huge combinatorial space of video format knobs; ii) the complex impacts of these knobs and their high profiling cost; iii) optimizing for multiple resource types. It explores an idea called backward derivation of configuration: in the opposite direction along the video data path, VStore passes the video quantity and quality expected by analytics backward to retrieval, to storage, and to ingestion. In this process, VStore derives an optimal set of video formats, optimizing for different resources in a progressive manner. VStore automatically derives large, complex configurations consisting of more than one hundred knobs over tens of video formats. In response to queries, VStore selects video formats catering to the executed operators and the target accuracy. It streams video data from disks through decoder to operators. It runs queries as fast as 362x of video realtime.