MOSIV: Multi-Object System Identification from Videos

TL;DR

MOSIV is a novel framework that enables accurate multi-object system identification from videos by optimizing continuous material parameters with a differentiable simulator, significantly advancing the understanding of complex multi-object interactions.

Contribution

Introduces MOSIV, the first method for multi-object system identification from videos using continuous parameters and geometric objectives, along with a synthetic benchmark for evaluation.

Findings

MOSIV improves grounding accuracy over baselines.

MOSIV achieves higher long-horizon simulation fidelity.

Object-level supervision is crucial for stable optimization.

Abstract

We introduce the challenging problem of multi-object system identification from videos, for which prior methods are ill-suited due to their focus on single-object scenes or discrete material classification with a fixed set of material prototypes. To address this, we propose MOSIV, a new framework that directly optimizes for continuous, per-object material parameters using a differentiable simulator guided by geometric objectives derived from video. We also present a new synthetic benchmark with contact-rich, multi-object interactions to facilitate evaluation. On this benchmark, MOSIV substantially improves grounding accuracy and long-horizon simulation fidelity over adapted baselines, establishing it as a strong baseline for this new task. Our analysis shows that object-level fine-grained supervision and geometry-aligned objectives are critical for stable optimization in these complex, multi-object settings. The source code and dataset will be released.