PhysX-3D: Physical-Grounded 3D Asset Generation

TL;DR

PhysX-3D introduces a comprehensive framework for generating 3D assets that incorporate physical properties, addressing a key gap in existing models and enabling more realistic and applicable 3D asset creation.

Contribution

The paper presents PhysXNet, a novel physics-grounded 3D dataset with a scalable annotation pipeline, and PhysXGen, a framework for physics-aware 3D asset generation from images.

Findings

PhysXNet provides extensive physics annotations for 3D assets.

PhysXGen effectively integrates physical knowledge into 3D generation.

Framework demonstrates superior performance and generalization in experiments.

Abstract

3D modeling is moving from virtual to physical. Existing 3D generation primarily emphasizes geometries and textures while neglecting physical-grounded modeling. Consequently, despite the rapid development of 3D generative models, the synthesized 3D assets often overlook rich and important physical properties, hampering their real-world application in physical domains like simulation and embodied AI. As an initial attempt to address this challenge, we propose \textbf{PhysX-3D}, an end-to-end paradigm for physical-grounded 3D asset generation. 1) To bridge the critical gap in physics-annotated 3D datasets, we present PhysXNet - the first physics-grounded 3D dataset systematically annotated across five foundational dimensions: absolute scale, material, affordance, kinematics, and function description. In particular, we devise a scalable human-in-the-loop annotation pipeline based on vision-language models, which enables efficient creation of physics-first assets from raw 3D assets.2) Furthermore, we propose \textbf{PhysXGen}, a feed-forward framework for physics-grounded image-to-3D asset generation, injecting physical knowledge into the pre-trained 3D structural space. Specifically, PhysXGen employs a dual-branch architecture to explicitly model the latent correlations between 3D structures and physical properties, thereby producing 3D assets with plausible physical predictions while preserving the native geometry quality. Extensive experiments validate the superior performance and promising generalization capability of our framework. All the code, data, and models will be released to facilitate future research in generative physical AI.