HIMO: A New Benchmark for Full-Body Human Interacting with Multiple Objects

TL;DR

This paper introduces HIMO, a comprehensive large-scale dataset for full-body human interactions with multiple objects, along with novel tasks and models for detailed HOI synthesis and control.

Contribution

HIMO is the first large-scale dataset capturing multi-object human interactions with detailed annotations and temporal segments, enabling new HOI synthesis tasks and models.

Findings

Models generalize well to unseen objects and interactions.

Dual-branch diffusion model effectively synthesizes HOIs.

Auto-regressive pipeline ensures smooth transition between segments.

Abstract

Generating human-object interactions (HOIs) is critical with the tremendous advances of digital avatars. Existing datasets are typically limited to humans interacting with a single object while neglecting the ubiquitous manipulation of multiple objects. Thus, we propose HIMO, a large-scale MoCap dataset of full-body human interacting with multiple objects, containing 3.3K 4D HOI sequences and 4.08M 3D HOI frames. We also annotate HIMO with detailed textual descriptions and temporal segments, benchmarking two novel tasks of HOI synthesis conditioned on either the whole text prompt or the segmented text prompts as fine-grained timeline control. To address these novel tasks, we propose a dual-branch conditional diffusion model with a mutual interaction module for HOI synthesis. Besides, an auto-regressive generation pipeline is also designed to obtain smooth transitions between HOI segments. Experimental results demonstrate the generalization ability to unseen object geometries and temporal compositions.