Skip-Plan: Procedure Planning in Instructional Videos via Condensed Action Space Learning

TL;DR

Skip-Plan introduces a novel procedure planning method for instructional videos that simplifies long action sequences into reliable shorter sub-chains, improving performance by avoiding high-dimensional supervision and error accumulation.

Contribution

It proposes a chain model with skipping strategy to condense action space, enabling more reliable and efficient procedure planning in instructional videos.

Findings

Achieves state-of-the-art results on CrossTask and COIN benchmarks.

Effectively reduces error propagation in long action sequences.

Demonstrates robustness by skipping unreliable intermediate actions.

Abstract

In this paper, we propose Skip-Plan, a condensed action space learning method for procedure planning in instructional videos. Current procedure planning methods all stick to the state-action pair prediction at every timestep and generate actions adjacently. Although it coincides with human intuition, such a methodology consistently struggles with high-dimensional state supervision and error accumulation on action sequences. In this work, we abstract the procedure planning problem as a mathematical chain model. By skipping uncertain nodes and edges in action chains, we transfer long and complex sequence functions into short but reliable ones in two ways. First, we skip all the intermediate state supervision and only focus on action predictions. Second, we decompose relatively long chains into multiple short sub-chains by skipping unreliable intermediate actions. By this means, our model explores all sorts of reliable sub-relations within an action sequence in the condensed action space. Extensive experiments show Skip-Plan achieves state-of-the-art performance on the CrossTask and COIN benchmarks for procedure planning.