LEMON: Learning 3D Human-Object Interaction Relation from 2D Images

TL;DR

LEMON is a unified model that leverages correlations between humans and objects to better predict 3D interaction elements, advancing embodied AI understanding from 2D images.

Contribution

The paper introduces LEMON, a novel approach that jointly models 3D human-object interactions by exploiting inherent correlations, unlike prior isolated element prediction methods.

Findings

LEMON outperforms existing methods in predicting interaction elements.

The 3DIR dataset supports comprehensive evaluation of 3D interaction modeling.

Joint modeling reduces uncertainty in interaction prediction.

Abstract

Learning 3D human-object interaction relation is pivotal to embodied AI and interaction modeling. Most existing methods approach the goal by learning to predict isolated interaction elements, e.g., human contact, object affordance, and human-object spatial relation, primarily from the perspective of either the human or the object. Which underexploit certain correlations between the interaction counterparts (human and object), and struggle to address the uncertainty in interactions. Actually, objects' functionalities potentially affect humans' interaction intentions, which reveals what the interaction is. Meanwhile, the interacting humans and objects exhibit matching geometric structures, which presents how to interact. In light of this, we propose harnessing these inherent correlations between interaction counterparts to mitigate the uncertainty and jointly anticipate the above interaction elements in 3D space. To achieve this, we present LEMON (LEarning 3D huMan-Object iNteraction relation), a unified model that mines interaction intentions of the counterparts and employs curvatures to guide the extraction of geometric correlations, combining them to anticipate the interaction elements. Besides, the 3D Interaction Relation dataset (3DIR) is collected to serve as the test bed for training and evaluation. Extensive experiments demonstrate the superiority of LEMON over methods estimating each element in isolation.