WhyAct: Identifying Action Reasons in Lifestyle Vlogs

TL;DR

This paper introduces WhyAct, a dataset and multimodal model for automatically identifying reasons behind actions in lifestyle vlogs, combining visual and textual cues to understand human behavior in videos.

Contribution

The paper presents the first dataset of action reasons in lifestyle vlogs and a multimodal model that integrates visual and textual data for reason inference.

Findings

Created the WhyAct dataset with 1,077 annotated actions and reasons.

Developed a multimodal model that effectively combines visual and textual information.

Demonstrated improved accuracy in action reason identification over baseline methods.

Abstract

We aim to automatically identify human action reasons in online videos. We focus on the widespread genre of lifestyle vlogs, in which people perform actions while verbally describing them. We introduce and make publicly available the WhyAct dataset, consisting of 1,077 visual actions manually annotated with their reasons. We describe a multimodal model that leverages visual and textual information to automatically infer the reasons corresponding to an action presented in the video.