A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking

TL;DR

This paper introduces PL-Stitch, a self-supervised learning framework that leverages the temporal order of video frames using Plackett-Luce ranking to improve procedural activity recognition in videos.

Contribution

It proposes a novel self-supervised approach with probabilistic objectives based on Plackett-Luce model to enhance understanding of procedural workflows in videos.

Findings

Achieves +11.4 percentage points in surgical phase recognition accuracy.

Attains +5.7 percentage points in cooking action segmentation accuracy.

Outperforms existing methods across five surgical and cooking benchmarks.

Abstract

Procedural activities, ranging from routine cooking to complex surgical operations, are highly structured sequences of actions performed in a specific temporal order. Despite the success of current self-supervised learning (SSL) methods on static images and short clips, these models often overlook the underlying sequential structure of such activities. We expose this lack of procedural awareness with a motivating experiment: models pretrained on forward and time-reversed sequences produce highly similar features, confirming that their representations are blind to the underlying procedural order. To address this shortcoming, we propose PL-Stitch, a self-supervised framework that harnesses the inherent temporal order of video frames as a powerful supervisory signal. Our approach integrates two novel probabilistic objectives based on the Plackett-Luce (PL) model. The primary PL objective trains the model to sort sampled frames chronologically, compelling it to learn the global workflow progression. The secondary objective, a spatio-temporal jigsaw loss, complements the learning by capturing fine-grained, cross-frame object correspondences. Our approach consistently achieves superior performance across five surgical and cooking benchmarks. Specifically, PL-Stitch yields significant gains in surgical phase recognition (e.g., +11.4 pp in k-NN accuracy on Cholec80) and cooking action segmentation (e.g., +5.7 pp in linear probing accuracy on Breakfast), demonstrating its effectiveness for procedural video representation learning. Code and models are available at https://github.com/visurg-ai/PL-Stitch.