Masked Diffusion with Task-awareness for Procedure Planning in Instructional Videos

TL;DR

This paper introduces a masked diffusion model with task-aware attention for improved procedure planning in instructional videos, leveraging joint visual-text embeddings to focus on relevant actions and achieve state-of-the-art results.

Contribution

The paper proposes a novel masked diffusion approach that uses task-aware attention and enhanced visual-text embeddings to better handle large decision spaces in instructional video analysis.

Findings

Achieved state-of-the-art performance on multiple datasets.

Effectively concentrates on relevant action types during diffusion.

Improved task classification accuracy with joint visual-text embeddings.

Abstract

A key challenge with procedure planning in instructional videos lies in how to handle a large decision space consisting of a multitude of action types that belong to various tasks. To understand real-world video content, an AI agent must proficiently discern these action types (e.g., pour milk, pour water, open lid, close lid, etc.) based on brief visual observation. Moreover, it must adeptly capture the intricate semantic relation of the action types and task goals, along with the variable action sequences. Recently, notable progress has been made via the integration of diffusion models and visual representation learning to address the challenge. However, existing models employ rudimentary mechanisms to utilize task information to manage the decision space. To overcome this limitation, we introduce a simple yet effective enhancement - a masked diffusion model. The introduced mask acts akin to a task-oriented attention filter, enabling the diffusion/denoising process to concentrate on a subset of action types. Furthermore, to bolster the accuracy of task classification, we harness more potent visual representation learning techniques. In particular, we learn a joint visual-text embedding, where a text embedding is generated by prompting a pre-trained vision-language model to focus on human actions. We evaluate the method on three public datasets and achieve state-of-the-art performance on multiple metrics. Code is available at https://github.com/ffzzy840304/Masked-PDPP.