PhysID: Physics-based Interactive Dynamics from a Single-view Image

TL;DR

PhysID enables the creation of physics-based interactive dynamics from a single image using generative models and on-device physics engines, advancing mobile AR/VR experiences with minimal manual effort.

Contribution

This work introduces PhysID, a novel framework that generates interactive physics-based 3D models from a single image, reducing the need for expert intervention and enabling real-time mobile applications.

Findings

Effective zero-shot performance of MLLMs on diverse tasks

Successful 3D reconstruction with minimal manual input

Real-time, physics-based rendering on mobile devices

Abstract

Transforming static images into interactive experiences remains a challenging task in computer vision. Tackling this challenge holds the potential to elevate mobile user experiences, notably through interactive and AR/VR applications. Current approaches aim to achieve this either using pre-recorded video responses or requiring multi-view images as input. In this paper, we present PhysID, that streamlines the creation of physics-based interactive dynamics from a single-view image by leveraging large generative models for 3D mesh generation and physical property prediction. This significantly reduces the expertise required for engineering-intensive tasks like 3D modeling and intrinsic property calibration, enabling the process to be scaled with minimal manual intervention. We integrate an on-device physics-based engine for physically plausible real-time rendering with user interactions. PhysID represents a leap forward in mobile-based interactive dynamics, offering real-time, non-deterministic interactions and user-personalization with efficient on-device memory consumption. Experiments evaluate the zero-shot capabilities of various Multimodal Large Language Models (MLLMs) on diverse tasks and the performance of 3D reconstruction models. These results demonstrate the cohesive functioning of all modules within the end-to-end framework, contributing to its effectiveness.