Temporally Coherent Bayesian Models for Entity Discovery in Videos by Tracklet Clustering

TL;DR

This paper introduces Bayesian nonparametric models leveraging temporal coherence for entity discovery in videos through tracklet clustering, improving accuracy and enabling online processing with minimal false detections.

Contribution

The paper presents the first Bayesian nonparametric models for tracklet-level temporal coherence, extending CRP to TC-CRP and TC-CRF for improved entity discovery in videos.

Findings

Significant improvement in cluster purity and person coverage.

Models perform online with streaming videos and reject false detections.

Applicable to various entity types beyond persons.

Abstract

A video can be represented as a sequence of tracklets, each spanning 10-20 frames, and associated with one entity (eg. a person). The task of \emph{Entity Discovery} in videos can be naturally posed as tracklet clustering. We approach this task by leveraging \emph{Temporal Coherence}(TC): the fundamental property of videos that each tracklet is likely to be associated with the same entity as its temporal neighbors. Our major contributions are the first Bayesian nonparametric models for TC at tracklet-level. We extend Chinese Restaurant Process (CRP) to propose TC-CRP, and further to Temporally Coherent Chinese Restaurant Franchise (TC-CRF) to jointly model short temporal segments. On the task of discovering persons in TV serial videos without meta-data like scripts, these methods show considerable improvement in cluster purity and person coverage compared to state-of-the-art approaches to tracklet clustering. We represent entities with mixture components, and tracklets with vectors of very generic features, which can work for any type of entity (not necessarily person). The proposed methods can perform online tracklet clustering on streaming videos with little performance deterioration unlike existing approaches, and can automatically reject tracklets resulting from false detections. Finally we discuss entity-driven video summarization- where some temporal segments of the video are selected automatically based on the discovered entities.