SVIP: Sequence VerIfication for Procedures in Videos

TL;DR

This paper introduces SVIP, a new sequence verification task for videos that distinguishes similar actions with step-level variations, using novel datasets, metrics, and a transformer-based baseline.

Contribution

It proposes the first sequence verification task for procedures in videos, along with new datasets, a novel evaluation metric, and a transformer-based baseline method.

Findings

The transformer-based baseline outperforms existing action recognition methods.

The Weighted Distance Ratio metric effectively evaluates sequence similarity under step variations.

New datasets enable comprehensive evaluation of sequence verification methods.

Abstract

In this paper, we propose a novel sequence verification task that aims to distinguish positive video pairs performing the same action sequence from negative ones with step-level transformations but still conducting the same task. Such a challenging task resides in an open-set setting without prior action detection or segmentation that requires event-level or even frame-level annotations. To that end, we carefully reorganize two publicly available action-related datasets with step-procedure-task structure. To fully investigate the effectiveness of any method, we collect a scripted video dataset enumerating all kinds of step-level transformations in chemical experiments. Besides, a novel evaluation metric Weighted Distance Ratio is introduced to ensure equivalence for different step-level transformations during evaluation. In the end, a simple but effective baseline based on the transformer encoder with a novel sequence alignment loss is introduced to better characterize long-term dependency between steps, which outperforms other action recognition methods. Codes and data will be released.