Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels

TL;DR

Rekall is a library that allows users to specify and detect new video events by composing outputs of pre-trained models through a query language, enabling rapid and accurate event detection without training new models.

Contribution

The paper introduces Rekall, a novel library for compositional video event specification using existing models, facilitating quick and accurate detection of domain-specific events.

Findings

Domain experts can create effective queries in hours to detect new events.

Rekall achieves detection accuracy comparable or superior to learned models.

Novice users can learn to author queries within an hour.

Abstract

Many real-world video analysis applications require the ability to identify domain-specific events in video, such as interviews and commercials in TV news broadcasts, or action sequences in film. Unfortunately, pre-trained models to detect all the events of interest in video may not exist, and training new models from scratch can be costly and labor-intensive. In this paper, we explore the utility of specifying new events in video in a more traditional manner: by writing queries that compose outputs of existing, pre-trained models. To write these queries, we have developed Rekall, a library that exposes a data model and programming model for compositional video event specification. Rekall represents video annotations from different sources (object detectors, transcripts, etc.) as spatiotemporal labels associated with continuous volumes of spacetime in a video, and provides operators for composing labels into queries that model new video events. We demonstrate the use of Rekall in analyzing video from cable TV news broadcasts, films, static-camera vehicular video streams, and commercial autonomous vehicle logs. In these efforts, domain experts were able to quickly (in a few hours to a day) author queries that enabled the accurate detection of new events (on par with, and in some cases much more accurate than, learned approaches) and to rapidly retrieve video clips for human-in-the-loop tasks such as video content curation and training data curation. Finally, in a user study, novice users of Rekall were able to author queries to retrieve new events in video given just one hour of query development time.