Online Action Detection in Streaming Videos with Time Buffers

TL;DR

This paper introduces a new online action detection framework that accounts for broadcast delay in streaming videos, improving detection accuracy by utilizing a small buffer time rather than immediate prediction.

Contribution

It proposes a novel problem setting for online action detection that incorporates buffer time, along with a new detection framework tailored for this setting.

Findings

Significant accuracy improvements over existing models.

Effective use of buffer time enhances detection performance.

Validated on three standard benchmarks.

Abstract

We formulate the problem of online temporal action detection in live streaming videos, acknowledging one important property of live streaming videos that there is normally a broadcast delay between the latest captured frame and the actual frame viewed by the audience. The standard setting of the online action detection task requires immediate prediction after a new frame is captured. We illustrate that its lack of consideration of the delay is imposing unnecessary constraints on the models and thus not suitable for this problem. We propose to adopt the problem setting that allows models to make use of the small `buffer time' incurred by the delay in live streaming videos. We design an action start and end detection framework for this online with buffer setting with two major components: flattened I3D and window-based suppression. Experiments on three standard temporal action detection benchmarks under the proposed setting demonstrate the effectiveness of the proposed framework. We show that by having a suitable problem setting for this problem with wide-applications, we can achieve much better detection accuracy than off-the-shelf online action detection models.