MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks

TL;DR

This paper introduces MultiScript, a new benchmark for multimodal script learning from videos and text, enabling AI systems to generate and predict steps for everyday tasks across diverse domains.

Contribution

It presents a novel benchmark with two tasks for multimodal script learning and proposes knowledge-guided generative frameworks leveraging large language models.

Findings

Proposed frameworks outperform baseline models.

MultiScript covers over 6,655 tasks across 19 domains.

Significant improvement in script generation and step prediction accuracy.

Abstract

Automatically generating scripts (i.e. sequences of key steps described in text) from video demonstrations and reasoning about the subsequent steps are crucial to the modern AI virtual assistants to guide humans to complete everyday tasks, especially unfamiliar ones. However, current methods for generative script learning rely heavily on well-structured preceding steps described in text and/or images or are limited to a certain domain, resulting in a disparity with real-world user scenarios. To address these limitations, we present a new benchmark challenge -- MultiScript, with two new tasks on task-oriented multimodal script learning: (1) multimodal script generation, and (2) subsequent step prediction. For both tasks, the input consists of a target task name and a video illustrating what has been done to complete the target task, and the expected output is (1) a sequence of structured step descriptions in text based on the demonstration video, and (2) a single text description for the subsequent step, respectively. Built from WikiHow, MultiScript covers multimodal scripts in videos and text descriptions for over 6,655 human everyday tasks across 19 diverse domains. To establish baseline performance on MultiScript, we propose two knowledge-guided multimodal generative frameworks that incorporate the task-related knowledge prompted from large language models such as Vicuna. Experimental results show that our proposed approaches significantly improve over the competitive baselines.