InterMask: 3D Human Interaction Generation via Collaborative Masked Modeling

TL;DR

InterMask is a novel framework that uses collaborative masked modeling and discrete motion tokens to generate realistic, high-fidelity 3D human interactions from text descriptions, outperforming previous methods.

Contribution

The paper introduces InterMask, a new approach combining VQ-VAE and transformer-based masked modeling for improved 3D human interaction generation.

Findings

Achieves state-of-the-art FID scores on InterHuman and InterX datasets.

Produces diverse and high-fidelity human interaction sequences.

Supports reaction generation without additional fine-tuning.

Abstract

Generating realistic 3D human-human interactions from textual descriptions remains a challenging task. Existing approaches, typically based on diffusion models, often produce results lacking realism and fidelity. In this work, we introduce InterMask, a novel framework for generating human interactions using collaborative masked modeling in discrete space. InterMask first employs a VQ-VAE to transform each motion sequence into a 2D discrete motion token map. Unlike traditional 1D VQ token maps, it better preserves fine-grained spatio-temporal details and promotes spatial awareness within each token. Building on this representation, InterMask utilizes a generative masked modeling framework to collaboratively model the tokens of two interacting individuals. This is achieved by employing a transformer architecture specifically designed to capture complex spatio-temporal inter-dependencies. During training, it randomly masks the motion tokens of both individuals and learns to predict them. For inference, starting from fully masked sequences, it progressively fills in the tokens for both individuals. With its enhanced motion representation, dedicated architecture, and effective learning strategy, InterMask achieves state-of-the-art results, producing high-fidelity and diverse human interactions. It outperforms previous methods, achieving an FID of $5.154$ (vs $5.535$ of in2IN) on the InterHuman dataset and $0.399$ (vs $5.207$ of InterGen) on the InterX dataset. Additionally, InterMask seamlessly supports reaction generation without the need for model redesign or fine-tuning.