PhysTalk: Language-driven Real-time Physics in 3D Gaussian Scenes

TL;DR

PhysTalk enables real-time, physics-based 4D animations from text prompts by integrating large language models with 3D Gaussian Splatting, eliminating the need for offline optimization and mesh extraction.

Contribution

It introduces the first framework coupling 3D Gaussian Splatting with a physics simulator driven by language models, enabling interactive, physics-based 3D animations from open vocabulary prompts.

Findings

Real-time physics-based 4D animations from text prompts.

Open vocabulary interaction without mesh extraction.

Lightweight, train-free, and computationally efficient.

Abstract

Realistic visual simulations are omnipresent, yet their creation requires computing time, rendering, and expert animation knowledge. Open-vocabulary visual effects generation from text inputs emerges as a promising solution that can unlock immense creative potential. However, current pipelines lack both physical realism and effective language interfaces, requiring slow offline optimization. In contrast, PhysTalk takes a 3D Gaussian Splatting (3DGS) scene as input and translates arbitrary user prompts into real time, physics based, interactive 4D animations. A large language model (LLM) generates executable code that directly modifies 3DGS parameters through lightweight proxies and particle dynamics. Notably, PhysTalk is the first framework to couple 3DGS directly with a physics simulator without relying on time consuming mesh extraction. While remaining open vocabulary, this design enables interactive 3D Gaussian animation via collision aware, physics based manipulation of arbitrary, multi material objects. Finally, PhysTalk is train-free and computationally lightweight: this makes 4D animation broadly accessible and shifts these workflows from a "render and wait" paradigm toward an interactive dialogue with a modern, physics-informed pipeline.