Light Future: Multimodal Action Frame Prediction via InstructPix2Pix

TL;DR

This paper introduces a lightweight, multimodal visual prediction framework for robotic action forecasting that uses a fine-tuned InstructPix2Pix model to predict future frames based on a single image and textual instruction, outperforming existing methods in efficiency and accuracy.

Contribution

It pioneers the adaptation of InstructPix2Pix for multimodal future frame prediction in robotics, reducing computational costs and inference latency significantly.

Findings

Achieves higher SSIM and PSNR than state-of-the-art baselines.

Requires only a single image and text prompt for prediction.

Enables faster inference with lower GPU demands.

Abstract

Predicting future motion trajectories is a critical capability across domains such as robotics, autonomous systems, and human activity forecasting, enabling safer and more intelligent decision-making. This paper proposes a novel, efficient, and lightweight approach for robot action prediction, offering significantly reduced computational cost and inference latency compared to conventional video prediction models. Importantly, it pioneers the adaptation of the InstructPix2Pix model for forecasting future visual frames in robotic tasks, extending its utility beyond static image editing. We implement a deep learning-based visual prediction framework that forecasts what a robot will observe 100 frames (10 seconds) into the future, given a current image and a textual instruction. We repurpose and fine-tune the InstructPix2Pix model to accept both visual and textual inputs, enabling multimodal future frame prediction. Experiments on the RoboTWin dataset (generated based on real-world scenarios) demonstrate that our method achieves superior SSIM and PSNR compared to state-of-the-art baselines in robot action prediction tasks. Unlike conventional video prediction models that require multiple input frames, heavy computation, and slow inference latency, our approach only needs a single image and a text prompt as input. This lightweight design enables faster inference, reduced GPU demands, and flexible multimodal control, particularly valuable for applications like robotics and sports motion trajectory analytics, where motion trajectory precision is prioritized over visual fidelity.