Bag of Attributes for Video Event Retrieval

TL;DR

This paper introduces the Bag-of-Attributes (BoA) model for video representation, leveraging semantic features from deep neural networks to improve video event retrieval with compact, high-level feature vectors.

Contribution

The paper proposes a novel BoA model that encodes videos into semantic attribute vectors using pre-trained CNNs, enhancing retrieval performance and compactness.

Findings

BoA achieves comparable or better retrieval results than baselines.

BoA provides a more compact video representation.

Semantic features from CNNs improve video retrieval accuracy.

Abstract

In this paper, we present the Bag-of-Attributes (BoA) model for video representation aiming at video event retrieval. The BoA model is based on a semantic feature space for representing videos, resulting in high-level video feature vectors. For creating a semantic space, i.e., the attribute space, we can train a classifier using a labeled image dataset, obtaining a classification model that can be understood as a high-level codebook. This model is used to map low-level frame vectors into high-level vectors (e.g., classifier probability scores). Then, we apply pooling operations to the frame vectors to create the final bag of attributes for the video. In the BoA representation, each dimension corresponds to one category (or attribute) of the semantic space. Other interesting properties are: compactness, flexibility regarding the classifier, and ability to encode multiple semantic concepts in a single video representation. Our experiments considered the semantic space created by state-of-the-art convolutional neural networks pre-trained on 1000 object categories of ImageNet. Such deep neural networks were used to classify each video frame and then different coding strategies were used to encode the probability distribution from the softmax layer into a frame vector. Next, different pooling strategies were used to combine frame vectors in the BoA representation for a video. Results using BoA were comparable or superior to the baselines in the task of video event retrieval using the EVVE dataset, with the advantage of providing a much more compact representation.