Enhancing Visual Planning with Auxiliary Tasks and Multi-token Prediction

TL;DR

This paper introduces VideoPlan, a novel approach that enhances visual planning in videos by using auxiliary tasks and multi-token prediction, achieving state-of-the-art results on multiple datasets.

Contribution

It proposes Auxiliary Task Augmentation and Multi-token Prediction to improve large multimodal models for long-horizon visual planning tasks.

Findings

Achieves 7.3% and 3.4% improvements on COIN and CrossTask datasets.

Performs on par with state-of-the-art on Ego4D Long-term Action Anticipation.

Outperforms prior methods by effectively modeling structured action spaces.

Abstract

Visual Planning for Assistance (VPA) aims to predict a sequence of user actions required to achieve a specified goal based on a video showing the user's progress. Although recent advances in multimodal large language models (MLLMs) have shown promising results in video understanding, long-horizon visual planning remains a challenging problem. We identify two challenges in training large MLLMs for video-based planning tasks: (1) scarcity of procedural annotations, limiting the model's ability to learn procedural task dynamics effectively, and (2) inefficiency of next-token prediction objective to explicitly capture the structured action space for visual planning when compared to free-form, natural language. To tackle data scarcity, we introduce Auxiliary Task Augmentation. We design and train our model on auxiliary tasks relevant to long-horizon video-based planning (e.g., goal prediction) to augment the model's planning ability. To more explicitly model the structured action space unique to visual planning tasks, we leverage Multi-token Prediction, extending traditional next-token prediction by using multiple heads to predict multiple future tokens during training. Our approach, VideoPlan, achieves state-of-the-art VPA performance on the COIN and CrossTask datasets, surpassing prior methods by 7.3% and 3.4%, respectively, when predicting 3 future actions. We further extend our method to the challenging Ego4D Long-term Action Anticipation task, and show that it is on par with the state-of-the-art approaches despite not using specialized egocentric features. Code will be made available.