Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning

TL;DR

This paper introduces an online dense video captioning method using a novel autoregressive factorized decoding architecture that produces frequent, detailed, and temporally aligned captions without future frame access, improving efficiency and comprehensiveness.

Contribution

The authors propose a new autoregressive factorized decoding model for online dense video captioning, enabling real-time, localized, and detailed descriptions while reducing computational costs.

Findings

Outperforms existing offline and online methods in accuracy

Uses 20% less compute than comparable models

Produces more comprehensive and frequent captions

Abstract

Generating automatic dense captions for videos that accurately describe their contents remains a challenging area of research. Most current models require processing the entire video at once. Instead, we propose an efficient, online approach which outputs frequent, detailed and temporally aligned captions, without access to future frames. Our model uses a novel autoregressive factorized decoding architecture, which models the sequence of visual features for each time segment, outputting localized descriptions and efficiently leverages the context from the previous video segments. This allows the model to output frequent, detailed captions to more comprehensively describe the video, according to its actual local content, rather than mimic the training data. Second, we propose an optimization for efficient training and inference, which enables scaling to longer videos. Our approach shows excellent performance compared to both offline and online methods, and uses 20\% less compute. The annotations produced are much more comprehensive and frequent, and can further be utilized in automatic video tagging and in large-scale video data harvesting.