Saying What You're Looking For: Linguistics Meets Video Search

TL;DR

This paper introduces a linguistically-informed video search method that accurately retrieves clips matching complex natural-language queries by combining compositional semantics with object detection and tracking.

Contribution

It presents a novel approach that integrates compositional semantics with object tracking to improve video search for complex natural-language queries.

Findings

Successfully searched 141 queries involving people and horses.

Achieved accurate retrieval of video clips depicting complex interactions.

Demonstrated effectiveness on 10 Hollywood movies.

Abstract

We present an approach to searching large video corpora for video clips which depict a natural-language query in the form of a sentence. This approach uses compositional semantics to encode subtle meaning that is lost in other systems, such as the difference between two sentences which have identical words but entirely different meaning: "The person rode the horse} vs. \emph{The horse rode the person". Given a video-sentence pair and a natural-language parser, along with a grammar that describes the space of sentential queries, we produce a score which indicates how well the video depicts the sentence. We produce such a score for each video clip in a corpus and return a ranked list of clips. Furthermore, this approach addresses two fundamental problems simultaneously: detecting and tracking objects, and recognizing whether those tracks depict the query. Because both tracking and object detection are unreliable, this uses knowledge about the intended sentential query to focus the tracker on the relevant participants and ensures that the resulting tracks are described by the sentential query. While earlier work was limited to single-word queries which correspond to either verbs or nouns, we show how one can search for complex queries which contain multiple phrases, such as prepositional phrases, and modifiers, such as adverbs. We demonstrate this approach by searching for 141 queries involving people and horses interacting with each other in 10 full-length Hollywood movies.