MeshMamba: State Space Models for Articulated 3D Mesh Generation and Reconstruction

TL;DR

MeshMamba introduces a scalable neural network model utilizing state space models for detailed 3D articulated mesh generation and reconstruction, enabling high-resolution body mesh modeling with improved performance over prior methods.

Contribution

The paper presents MeshMamba, a novel approach that employs Mamba State Space Models for efficient, scalable 3D mesh learning, including new diffusion and recovery models for detailed human body meshes.

Findings

Outperforms previous methods in 3D human shape generation.

Enables whole-body mesh recovery with face and hands in near real-time.

Successfully reconstructs detailed meshes with over 10,000 vertices.

Abstract

In this paper, we introduce MeshMamba, a neural network model for learning 3D articulated mesh models by employing the recently proposed Mamba State Space Models (Mamba-SSMs). MeshMamba is efficient and scalable in handling a large number of input tokens, enabling the generation and reconstruction of body mesh models with more than 10,000 vertices, capturing clothing and hand geometries. The key to effectively learning MeshMamba is the serialization technique of mesh vertices into orderings that are easily processed by Mamba. This is achieved by sorting the vertices based on body part annotations or the 3D vertex locations of a template mesh, such that the ordering respects the structure of articulated shapes. Based on MeshMamba, we design 1) MambaDiff3D, a denoising diffusion model for generating 3D articulated meshes and 2) Mamba-HMR, a 3D human mesh recovery model that reconstructs a human body shape and pose from a single image. Experimental results showed that MambaDiff3D can generate dense 3D human meshes in clothes, with grasping hands, etc., and outperforms previous approaches in the 3D human shape generation task. Additionally, Mamba-HMR extends the capabilities of previous non-parametric human mesh recovery approaches, which were limited to handling body-only poses using around 500 vertex tokens, to the whole-body setting with face and hands, while achieving competitive performance in (near) real-time.