Event-Guided Procedure Planning from Instructional Videos with Text Supervision

TL;DR

This paper introduces an event-guided approach for procedure planning from instructional videos with text supervision, effectively bridging the semantic gap between visual states and actions by inferring events and planning actions accordingly.

Contribution

The paper proposes a novel event-guided paradigm and the E3P model that incorporates event inference and relation mining for improved procedure planning from videos.

Findings

E3P outperforms previous methods on three datasets.

Event inference improves planning accuracy.

Mask-and-predict enhances relation modeling.

Abstract

In this work, we focus on the task of procedure planning from instructional videos with text supervision, where a model aims to predict an action sequence to transform the initial visual state into the goal visual state. A critical challenge of this task is the large semantic gap between observed visual states and unobserved intermediate actions, which is ignored by previous works. Specifically, this semantic gap refers to that the contents in the observed visual states are semantically different from the elements of some action text labels in a procedure. To bridge this semantic gap, we propose a novel event-guided paradigm, which first infers events from the observed states and then plans out actions based on both the states and predicted events. Our inspiration comes from that planning a procedure from an instructional video is to complete a specific event and a specific event usually involves specific actions. Based on the proposed paradigm, we contribute an Event-guided Prompting-based Procedure Planning (E3P) model, which encodes event information into the sequential modeling process to support procedure planning. To further consider the strong action associations within each event, our E3P adopts a mask-and-predict approach for relation mining, incorporating a probabilistic masking scheme for regularization. Extensive experiments on three datasets demonstrate the effectiveness of our proposed model.