Joint Image-Text News Topic Detection and Tracking with And-Or Graph Representation

TL;DR

This paper introduces a hierarchical And-Or graph model for joint detection and tracking of news topics in broadcast videos, effectively capturing the structure of text and visuals to produce meaningful topic trajectories.

Contribution

It presents a novel hierarchical And-Or graph representation combined with cluster sampling and SWC for improved news topic detection and tracking in multimedia data.

Findings

Outperforms state-of-the-art methods on Reuters-21578 dataset

Explicitly models textual and visual data in news videos

Generates meaningful topic trajectories over time

Abstract

In this paper, we aim to develop a method for automatically detecting and tracking topics in broadcast news. We present a hierarchical And-Or graph (AOG) to jointly represent the latent structure of both texts and visuals. The AOG embeds a context sensitive grammar that can describe the hierarchical composition of news topics by semantic elements about people involved, related places and what happened, and model contextual relationships between elements in the hierarchy. We detect news topics through a cluster sampling process which groups stories about closely related events. Swendsen-Wang Cuts (SWC), an effective cluster sampling algorithm, is adopted for traversing the solution space and obtaining optimal clustering solutions by maximizing a Bayesian posterior probability. Topics are tracked to deal with the continuously updated news streams. We generate topic trajectories to show how topics emerge, evolve and disappear over time. The experimental results show that our method can explicitly describe the textual and visual data in news videos and produce meaningful topic trajectories. Our method achieves superior performance compared to state-of-the-art methods on both a public dataset Reuters-21578 and a self-collected dataset named UCLA Broadcast News Dataset.