Event and Entity Extraction from Generated Video Captions

TL;DR

This paper presents a framework for extracting semantic metadata such as entities, properties, relations, and video categories from automatically generated video captions, aiming to reduce manual annotation efforts.

Contribution

It introduces a method to extract detailed semantic metadata from captions generated by state-of-the-art dense video captioning models, linking caption quality to metadata extraction accuracy.

Findings

Metadata extraction is feasible from generated captions.

Extraction quality depends on event localization accuracy.

Caption generation performance influences metadata quality.

Abstract

Annotation of multimedia data by humans is time-consuming and costly, while reliable automatic generation of semantic metadata is a major challenge. We propose a framework to extract semantic metadata from automatically generated video captions. As metadata, we consider entities, the entities' properties, relations between entities, and the video category. We employ two state-of-the-art dense video captioning models with masked transformer (MT) and parallel decoding (PVDC) to generate captions for videos of the ActivityNet Captions dataset. Our experiments show that it is possible to extract entities, their properties, relations between entities, and the video category from the generated captions. We observe that the quality of the extracted information is mainly influenced by the quality of the event localization in the video as well as the performance of the event caption generation.