MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding

TL;DR

MMHU is a large-scale, multimodal benchmark dataset designed for comprehensive human behavior understanding in autonomous driving, including motion, intention, and safety-related behaviors, with diverse annotations and multiple evaluation tasks.

Contribution

The paper introduces MMHU, a novel extensive benchmark dataset with rich annotations for human behavior analysis in driving scenarios, covering multiple tasks and sources.

Findings

Dataset includes 57k motion clips and 1.73M frames.

Benchmark results across various tasks demonstrate the dataset's utility.

Rich annotations enable diverse behavior understanding tasks.

Abstract

Humans are integral components of the transportation ecosystem, and understanding their behaviors is crucial to facilitating the development of safe driving systems. Although recent progress has explored various aspects of human behavior$\unicode{x2014}$such as motion, trajectories, and intention$\unicode{x2014}$a comprehensive benchmark for evaluating human behavior understanding in autonomous driving remains unavailable. In this work, we propose $\textbf{MMHU}$, a large-scale benchmark for human behavior analysis featuring rich annotations, such as human motion and trajectories, text description for human motions, human intention, and critical behavior labels relevant to driving safety. Our dataset encompasses 57k human motion clips and 1.73M frames gathered from diverse sources, including established driving datasets such as Waymo, in-the-wild videos from YouTube, and self-collected data. A human-in-the-loop annotation pipeline is developed to generate rich behavior captions. We provide a thorough dataset analysis and benchmark multiple tasks$\unicode{x2014}$ranging from motion prediction to motion generation and human behavior question answering$\unicode{x2014}$thereby offering a broad evaluation suite. Project page : https://MMHU-Benchmark.github.io.