Future Optical Flow Prediction Improves Robot Control & Video Generation

TL;DR

FOFPred is a novel language-conditioned optical flow forecasting model that combines a Vision-Language Model and Diffusion architecture, trained on web-scale data, to improve control and video generation tasks.

Contribution

It introduces a unified VLM-Diffusion architecture for future optical flow prediction, trained on noisy web data, enabling strong multimodal reasoning and generalization.

Findings

Effective in robotic manipulation tasks

Enhances video generation quality

Demonstrates cross-domain versatility

Abstract

Future motion representations, such as optical flow, offer immense value for control and generative tasks. However, forecasting generalizable spatially dense motion representations remains a key challenge, and learning such forecasting from noisy, real-world data remains relatively unexplored. We introduce FOFPred, a novel language-conditioned optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture. This unique combination enables strong multimodal reasoning with pixel-level generative fidelity for future motion prediction. Our model is trained on web-scale human activity data-a highly scalable but unstructured source. To extract meaningful signals from this noisy video-caption data, we employ crucial data preprocessing techniques and our unified architecture with strong image pretraining. The resulting trained model is then extended to tackle two distinct downstream tasks in control and generation. Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred, confirming the value of a unified VLM-Diffusion architecture and scalable learning from diverse web data for future optical flow prediction.