Learning to Generate Human-Human-Object Interactions from Textual Descriptions

TL;DR

This paper introduces a novel framework for generating realistic human-human-object interactions from text, including a new dataset, synthesis methods, and a unified generative model that handles multi-human scenarios.

Contribution

It presents the first comprehensive approach to model and generate multi-human interactions with objects from textual descriptions, including dataset creation and a unified diffusion-based generative framework.

Findings

Outperforms previous single-human HOI generation methods

Successfully synthesizes multi-human interactions involving objects

Generates realistic and contextually appropriate HHOIs from text

Abstract

The way humans interact with each other, including interpersonal distances, spatial configuration, and motion, varies significantly across different situations. To enable machines to understand such complex, context-dependent behaviors, it is essential to model multiple people in relation to the surrounding scene context. In this paper, we present a novel research problem to model the correlations between two people engaged in a shared interaction involving an object. We refer to this formulation as Human-Human-Object Interactions (HHOIs). To overcome the lack of dedicated datasets for HHOIs, we present a newly captured HHOIs dataset and a method to synthesize HHOI data by leveraging image generative models. As an intermediary, we obtain individual human-object interaction (HOIs) and human-human interaction (HHIs) from the HHOIs, and with these data, we train an text-to-HOI and text-to-HHI model using score-based diffusion model. Finally, we present a unified generative framework that integrates the two individual model, capable of synthesizing complete HHOIs in a single advanced sampling process. Our method extends HHOI generation to multi-human settings, enabling interactions involving more than two individuals. Experimental results show that our method generates realistic HHOIs conditioned on textual descriptions, outperforming previous approaches that focus only on single-human HOIs. Furthermore, we introduce multi-human motion generation involving objects as an application of our framework.