Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks

TL;DR

This paper introduces a regularized deep neural network framework that leverages feature and class relationships to enhance video categorization accuracy, outperforming existing methods on multiple benchmarks.

Contribution

The paper proposes a novel regularized DNN that jointly exploits feature and class relationships, improving semantic modeling in video categorization tasks.

Findings

rDNN achieves state-of-the-art performance on Hollywood2 and Columbia datasets.

The approach is efficiently implemented on GPU with affordable training costs.

A new large-scale benchmark dataset, FCVID, is introduced for future research.

Abstract

In this paper, we study the challenging problem of categorizing videos according to high-level semantics such as the existence of a particular human action or a complex event. Although extensive efforts have been devoted in recent years, most existing works combined multiple video features using simple fusion strategies and neglected the utilization of inter-class semantic relationships. This paper proposes a novel unified framework that jointly exploits the feature relationships and the class relationships for improved categorization performance. Specifically, these two types of relationships are estimated and utilized by rigorously imposing regularizations in the learning process of a deep neural network (DNN). Such a regularized DNN (rDNN) can be efficiently realized using a GPU-based implementation with an affordable training cost. Through arming the DNN with better capability of harnessing both the feature and the class relationships, the proposed rDNN is more suitable for modeling video semantics. With extensive experimental evaluations, we show that rDNN produces superior performance over several state-of-the-art approaches. On the well-known Hollywood2 and Columbia Consumer Video benchmarks, we obtain very competitive results: 66.9\% and 73.5\% respectively in terms of mean average precision. In addition, to substantially evaluate our rDNN and stimulate future research on large scale video categorization, we collect and release a new benchmark dataset, called FCVID, which contains 91,223 Internet videos and 239 manually annotated categories.