Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference

TL;DR

This paper presents a new multimodal dataset combining videos and language with human actions, designed to evaluate logical inference systems involving complex semantic structures like negation and quantification.

Contribution

The paper introduces a novel dataset with videos, action labels, and logical triplets for multimodal inference, focusing on dynamic human actions and complex language expressions.

Findings

Dataset contains 200 videos and 5,554 action labels.

Includes 1,942 action triplets with logical semantic representations.

Facilitates evaluation of systems handling negation and quantification.

Abstract

This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form <subject, predicate, object> that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.