A Large-Scale Multimodal Dataset and Benchmarks for Human Activity Scene Understanding and Reasoning

TL;DR

This paper introduces CUHK-X, a large-scale multimodal dataset with benchmarks for human activity recognition, understanding, and reasoning, addressing the lack of fine-grained, multimodal data for advancing LLMs and LVLMs in complex activity analysis.

Contribution

The paper presents CUHK-X, a new multimodal dataset with scene creation methods and benchmarks for HAU and HARn, enabling improved LLM and LVLM applications in activity understanding.

Findings

Achieved 76.52% accuracy in HAR

Achieved 40.76% accuracy in HAU

Achieved 70.25% accuracy in HARn

Abstract

Multimodal human action recognition (HAR) leverages complementary sensors for activity classification. Beyond recognition, recent advances in large language models (LLMs) enable detailed descriptions and causal reasoning, motivating new tasks: human action understanding (HAU) and human action reasoning (HARn). However, most LLMs, especially large vision language models (LVLMs), struggle with non-RGB modalities such as depth, IMU, and mmWave due to the lack of large-scale data-caption resources. Existing HAR datasets mainly provide coarse data-label annotations, which are insufficient to capture fine-grained action dynamics needed for HAU and HARn. We consider two ground-truth pair types: (1) data label (discrete category) and (2) data caption (textual description). Naively generating captions from labels often lacks logical and spatiotemporal consistency. We introduce CUHK-X, a large-scale multimodal dataset and benchmark suite for HAR, HAU, and HARn. CUHK-X contains 58,445 samples covering 40 actions performed by 30 participants across two indoor environments. To improve caption consistency, we propose a prompt-based scene creation method that leverages LLMs to generate logically connected activity sequences, followed by human validation. CUHK-X includes three benchmarks with six evaluation tasks. Experiments report average accuracies of 76.52% (HAR), 40.76% (HAU), and 70.25% (HARn). CUHK-X aims to enable the community to apply and develop data-intensive learning methods for robust, multimodal human activity analysis. Project page and code: https://openaiotlab.github.io/CUHK-X/ and https://github.com/openaiotlab/CUHK-X.