Pixie: Fast and Generalizable Supervised Learning of 3D Physics from Pixels

TL;DR

Pixie introduces a fast, generalizable neural network that predicts physical properties of 3D scenes from visual data, enabling realistic physics simulation and zero-shot generalization to real-world scenes.

Contribution

The paper presents PIXIE, a novel supervised learning approach that predicts 3D scene physics from images, trained on a large dataset, and capable of fast inference and zero-shot generalization.

Findings

PIXIE outperforms test-time optimization methods by 1.46-4.39x in accuracy.

PIXIE is orders of magnitude faster than existing methods.

Using pretrained features like CLIP enables zero-shot generalization to real scenes.

Abstract

Inferring the physical properties of 3D scenes from visual information is a critical yet challenging task for creating interactive and realistic virtual worlds. While humans intuitively grasp material characteristics such as elasticity or stiffness, existing methods often rely on slow, per-scene optimization, limiting their generalizability and application. To address this problem, we introduce PIXIE, a novel method that trains a generalizable neural network to predict physical properties across multiple scenes from 3D visual features purely using supervised losses. Once trained, our feed-forward network can perform fast inference of plausible material fields, which coupled with a learned static scene representation like Gaussian Splatting enables realistic physics simulation under external forces. To facilitate this research, we also collected PIXIEVERSE, one of the largest known datasets of paired 3D assets and physic material annotations. Extensive evaluations demonstrate that PIXIE is about 1.46-4.39x better and orders of magnitude faster than test-time optimization methods. By leveraging pretrained visual features like CLIP, our method can also zero-shot generalize to real-world scenes despite only ever been trained on synthetic data. https://pixie-3d.github.io/