PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks

TL;DR

PlaTe introduces a transformer-based model that learns structured, goal-directed planning from instructional videos, effectively handling appearance gaps and reducing decision errors in procedural tasks.

Contribution

The paper presents PlaTe, a novel transformer-based approach that learns latent state-action representations from videos, improving long-term planning in procedural tasks.

Findings

Outperforms previous algorithms in goal-reaching tasks

Successfully applies to real-world instructional videos and interactive environments

Demonstrates feasibility on a UR-5 robotic platform

Abstract

In this work, we study the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the ability to plan a goal-directed procedure from real-world videos. Learning structured and plannable state and action spaces directly from unstructured videos is the key technical challenge of our task. There are two problems: first, the appearance gap between the training and validation datasets could be large for unstructured videos; second, these gaps lead to decision errors that compound over the steps. We address these limitations with Planning Transformer (PlaTe), which has the advantage of circumventing the compounding prediction errors that occur with single-step models during long model-based rollouts. Our method simultaneously learns the latent state and action information of assigned tasks and the representations of the decision-making process from human demonstrations. Experiments conducted on real-world instructional videos and an interactive environment show that our method can achieve a better performance in reaching the indicated goal than previous algorithms. We also validated the possibility of applying procedural tasks on a UR-5 platform. We make our code publicly available and support academic research purposes.