Masked Temporal Interpolation Diffusion for Procedure Planning in Instructional Videos

TL;DR

This paper introduces the Masked Temporal Interpolation Diffusion (MTID) model, which enhances procedure planning in instructional videos by generating coherent action sequences through a novel latent space interpolation and action-aware mechanisms.

Contribution

The paper presents a new diffusion-based model with a latent space temporal interpolation module and action-aware mask projection for improved task-specific procedure planning.

Findings

Achieves superior action planning accuracy on benchmark datasets.

Effectively captures intricate temporal relationships among actions.

Enables end-to-end training with richer mid-state supervision.

Abstract

In this paper, we address the challenge of procedure planning in instructional videos, aiming to generate coherent and task-aligned action sequences from start and end visual observations. Previous work has mainly relied on text-level supervision to bridge the gap between observed states and unobserved actions, but it struggles with capturing intricate temporal relationships among actions. Building on these efforts, we propose the Masked Temporal Interpolation Diffusion (MTID) model that introduces a latent space temporal interpolation module within the diffusion model. This module leverages a learnable interpolation matrix to generate intermediate latent features, thereby augmenting visual supervision with richer mid-state details. By integrating this enriched supervision into the model, we enable end-to-end training tailored to task-specific requirements, significantly enhancing the model's capacity to predict temporally coherent action sequences. Additionally, we introduce an action-aware mask projection mechanism to restrict the action generation space, combined with a task-adaptive masked proximity loss to prioritize more accurate reasoning results close to the given start and end states over those in intermediate steps. Simultaneously, it filters out task-irrelevant action predictions, leading to contextually aware action sequences. Experimental results across three widely used benchmark datasets demonstrate that our MTID achieves promising action planning performance on most metrics. The code is available at https://github.com/WiserZhou/MTID.