Optimizing Latency for Online Video CaptioningUsing Audio-Visual Transformers

TL;DR

This paper introduces a novel audio-visual Transformer-based method for low-latency online video captioning, enabling early caption generation with high quality by optimizing timing based on event detection and partial video frames.

Contribution

It proposes a joint training approach of a Transformer and timing detector to produce accurate captions early in video streams, reducing latency significantly.

Findings

Achieves 94% caption quality using only 28% of initial frames

Enables early captioning triggered by event detection or prediction

Outperforms traditional methods in latency-accuracy trade-off

Abstract

Video captioning is an essential technology to understand scenes and describe events in natural language. To apply it to real-time monitoring, a system needs not only to describe events accurately but also to produce the captions as soon as possible. Low-latency captioning is needed to realize such functionality, but this research area for online video captioning has not been pursued yet. This paper proposes a novel approach to optimize each caption's output timing based on a trade-off between latency and caption quality. An audio-visual Trans-former is trained to generate ground-truth captions using only a small portion of all video frames, and to mimic outputs of a pre-trained Transformer to which all the frames are given. A CNN-based timing detector is also trained to detect a proper output timing, where the captions generated by the two Trans-formers become sufficiently close to each other. With the jointly trained Transformer and timing detector, a caption can be generated in the early stages of an event-triggered video clip, as soon as an event happens or when it can be forecasted. Experiments with the ActivityNet Captions dataset show that our approach achieves 94% of the caption quality of the upper bound given by the pre-trained Transformer using the entire video clips, using only 28% of frames from the beginning.