Chain of Time: In-Context Physical Simulation with Image Generation Models

TL;DR

This paper introduces 'Chain of Time', a method that enhances physical reasoning in vision-language models by generating intermediate images during simulation, improving understanding of physical properties without additional training.

Contribution

It presents a novel, cognitively-inspired inference technique that improves physical simulation in image generation models and provides new insights into their reasoning capabilities.

Findings

Significantly improves physical simulation performance in image models

Reveals models' ability to simulate physical properties over time

Identifies limitations in models' inference of physical parameters

Abstract

We propose a novel cognitively-inspired method to improve and interpret physical simulation in vision-language models. Our ``Chain of Time" method involves generating a series of intermediate images during a simulation, and it is motivated by in-context reasoning in machine learning, as well as mental simulation in humans. Chain of Time is used at inference time, and requires no additional fine-tuning. We apply the Chain-of-Time method to synthetic and real-world domains, including 2-D graphics simulations and natural 3-D videos. These domains test a variety of particular physical properties, including velocity, acceleration, fluid dynamics, and conservation of momentum. We found that using Chain-of-Time simulation substantially improves the performance of a state-of-the-art image generation model. Beyond examining performance, we also analyzed the specific states of the world simulated by an image model at each time step, which sheds light on the dynamics underlying these simulations. This analysis reveals insights that are hidden from traditional evaluations of physical reasoning, including cases where an image generation model is able to simulate physical properties that unfold over time, such as velocity, gravity, and collisions. Our analysis also highlights particular cases where the image generation model struggles to infer particular physical parameters from input images, despite being capable of simulating relevant physical processes.