ViLPAct: A Benchmark for Compositional Generalization on Multimodal Human Activities

TL;DR

ViLPAct is a new multimodal benchmark dataset designed to evaluate AI agents' ability to reason and predict human activities based on videos and text, highlighting challenges in compositional generalization.

Contribution

The paper introduces ViLPAct, a comprehensive multimodal benchmark with datasets, test sets, and baseline models for human activity planning tasks.

Findings

Challenges in compositional generalization identified

Multimodal information fusion remains difficult

Baseline models show room for improvement

Abstract

We introduce ViLPAct, a novel vision-language benchmark for human activity planning. It is designed for a task where embodied AI agents can reason and forecast future actions of humans based on video clips about their initial activities and intents in text. The dataset consists of 2.9k videos from \charades extended with intents via crowdsourcing, a multi-choice question test set, and four strong baselines. One of the baselines implements a neurosymbolic approach based on a multi-modal knowledge base (MKB), while the other ones are deep generative models adapted from recent state-of-the-art (SOTA) methods. According to our extensive experiments, the key challenges are compositional generalization and effective use of information from both modalities.