Bootstrapping Action-Grounded Visual Dynamics in Unified Vision-Language Models

TL;DR

This paper explores how vision-language models can predict future visual states from actions, proposing a novel approach using inverse dynamics prediction to bootstrap forward dynamics prediction, leading to improved image editing performance.

Contribution

It introduces a method to leverage inverse dynamics prediction for bootstrapping forward dynamics prediction in vision-language models, enhancing their ability to generate plausible future frames.

Findings

Achieved competitive image editing performance with state-of-the-art models.

Improved image editing accuracy by 7-13% over existing methods.

Obtained the best human evaluation scores across Aurora-Bench subsets.

Abstract

Can unified vision-language models (VLMs) perform forward dynamics prediction (FDP), i.e., predicting the future state (in image form) given the previous observation and an action (in language form)? We find that VLMs struggle to generate physically plausible transitions between frames from instructions. Nevertheless, we identify a crucial asymmetry in multimodal grounding: fine-tuning a VLM to learn inverse dynamics prediction (IDP), effectively captioning the action between frames, is significantly easier than learning FDP. In turn, IDP can be used to bootstrap FDP through two main strategies: 1) weakly supervised learning from synthetic data and 2) inference time verification. Firstly, IDP can annotate actions for unlabelled pairs of video frame observations to expand the training data scale for FDP. Secondly, IDP can assign rewards to multiple samples of FDP to score them, effectively guiding search at inference time. We evaluate the FDP resulting from both strategies through the task of action-centric image editing on Aurora-Bench with two families of VLMs. Despite remaining general-purpose, our best model achieves a performance competitive with state-of-the-art image editing models, improving on them by a margin between $7\%$ and $13\%$ according to GPT4o-as-judge, and achieving the best average human evaluation across all subsets of Aurora-Bench.