InterDyn: Controllable Interactive Dynamics with Video Diffusion Models

TL;DR

InterDyn introduces a framework that uses large video diffusion models as implicit physics simulators to generate controllable, realistic videos of object interactions, surpassing static prediction methods and generalizing to new objects.

Contribution

It presents a novel method leveraging video diffusion models for interactive dynamics prediction with controllable, continuous motion generation, addressing limitations of prior static state transition approaches.

Findings

Generates plausible, temporally consistent interaction videos.

Outperforms static prediction baselines in quantitative evaluations.

Generalizes well to unseen objects and interactions.

Abstract

Predicting the dynamics of interacting objects is essential for both humans and intelligent systems. However, existing approaches are limited to simplified, toy settings and lack generalizability to complex, real-world environments. Recent advances in generative models have enabled the prediction of state transitions based on interventions, but focus on generating a single future state which neglects the continuous dynamics resulting from the interaction. To address this gap, we propose InterDyn, a novel framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor. Our key insight is that large video generation models can act as both neural renderers and implicit physics ``simulators'', having learned interactive dynamics from large-scale video data. To effectively harness this capability, we introduce an interactive control mechanism that conditions the video generation process on the motion of the driving entity. Qualitative results demonstrate that InterDyn generates plausible, temporally consistent videos of complex object interactions while generalizing to unseen objects. Quantitative evaluations show that InterDyn outperforms baselines that focus on static state transitions. This work highlights the potential of leveraging video generative models as implicit physics engines. Project page: https://interdyn.is.tue.mpg.de/