Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video

TL;DR

This paper introduces a dynamic feature prioritization method for activity detection in streaming videos, enabling timely and resource-efficient predictions by learning to schedule feature computation adaptively.

Contribution

It proposes a novel active approach that dynamically schedules feature computation based on video context, improving accuracy under computational constraints.

Findings

Significantly outperforms static methods across datasets.

Effective in both batch and streaming scenarios.

Adapts to computational budgets for timely predictions.

Abstract

Current approaches for activity recognition often ignore constraints on computational resources: 1) they rely on extensive feature computation to obtain rich descriptors on all frames, and 2) they assume batch-mode access to the entire test video at once. We propose a new active approach to activity recognition that prioritizes "what to compute when" in order to make timely predictions. The main idea is to learn a policy that dynamically schedules the sequence of features to compute on selected frames of a given test video. In contrast to traditional static feature selection, our approach continually re-prioritizes computation based on the accumulated history of observations and accounts for the transience of those observations in ongoing video. We develop variants to handle both the batch and streaming settings. On two challenging datasets, our method provides significantly better accuracy than alternative techniques for a wide range of computational budgets.