Learning from Multiple Sources for Video Summarisation

TL;DR

This paper introduces an unsupervised multi-source learning framework that combines visual and non-visual data to improve video summarisation and content understanding in surveillance videos.

Contribution

It proposes a novel method to jointly learn from heterogeneous visual and non-visual data sources, handling discrepancies and missing data effectively.

Findings

Improved video content clustering over state-of-the-art methods.

Accurate inference of missing non-visual semantics in unseen videos.

Validated video summarisation quality through user study.

Abstract

Many visual surveillance tasks, e.g.video summarisation, is conventionally accomplished through analysing imagerybased features. Relying solely on visual cues for public surveillance video understanding is unreliable, since visual observations obtained from public space CCTV video data are often not sufficiently trustworthy and events of interest can be subtle. On the other hand, non-visual data sources such as weather reports and traffic sensory signals are readily accessible but are not explored jointly to complement visual data for video content analysis and summarisation. In this paper, we present a novel unsupervised framework to learn jointly from both visual and independently-drawn non-visual data sources for discovering meaningful latent structure of surveillance video data. In particular, we investigate ways to cope with discrepant dimension and representation whist associating these heterogeneous data sources, and derive effective mechanism to tolerate with missing and incomplete data from different sources. We show that the proposed multi-source learning framework not only achieves better video content clustering than state-of-the-art methods, but also is capable of accurately inferring missing non-visual semantics from previously unseen videos. In addition, a comprehensive user study is conducted to validate the quality of video summarisation generated using the proposed multi-source model.