Visual Goal-Step Inference using wikiHow

TL;DR

This paper introduces the Visual Goal-Step Inference (VGSI) task, which involves selecting images that depict plausible steps towards a goal, using a large wikiHow dataset to challenge and improve multimodal reasoning models.

Contribution

The paper presents the VGSI task, a new dataset from wikiHow, and demonstrates its utility in training models for multimodal reasoning about procedural events.

Findings

VGSI is challenging for current models

Transfer learning improves VGSI accuracy by 15-20%

The dataset enables better multimodal reasoning about actions

Abstract

Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.