GenMatter: Perceiving Physical Objects with Generative Matter Models

TL;DR

This paper introduces GenMatter, a generative model inspired by human perception that hierarchically groups motion cues and appearance features to identify and track physical objects across diverse visual settings.

Contribution

The authors propose a unified, hardware-accelerated generative framework that effectively perceives and segments moving objects in various visual environments, bridging gaps in existing computer vision methods.

Findings

Captures human-like object perception in random dot kinematograms.

Recovers 3D structure and accurate segmentation from camouflaged objects.

Tracks deforming objects in natural RGB videos for scene understanding.

Abstract

Human visual perception offers valuable insights for understanding computational principles of motion-based scene interpretation. Humans robustly detect and segment moving entities that constitute independently moveable chunks of matter, whether observing sparse moving dots, textured surfaces, or naturalistic scenes. In contrast, existing computer vision systems lack a unified approach that works across these diverse settings. Inspired by principles of human perception, we propose a generative model that hierarchically groups low-level motion cues and high-level appearance features into particles (small Gaussians representing local matter), and groups particles into clusters capturing coherently and independently moveable physical entities. We develop a hardware-accelerated inference algorithm based on parallelized block Gibbs sampling to recover stable particle motion and groupings. Our model operates on different kinds of inputs (random dots, stylized textures, or naturalistic RGB video), enabling it to work across settings where biological vision succeeds but existing computer vision approaches do not. We validate this unified framework across three domains: on 2D random dot kinematograms, our approach captures human object perception including graded uncertainty across ambiguous conditions; on a Gestalt-inspired dataset of camouflaged rotating objects, our approach recovers correct 3D structure from motion and thereby accurate 2D object segmentation; and on naturalistic RGB videos, our model tracks the moving 3D matter that makes up deforming objects, enabling robust object-level scene understanding. This work thus establishes a general framework for motion-based perception grounded in principles of human vision.