Video Generation Models in Robotics -- Applications, Research Challenges, Future Directions

TL;DR

This survey reviews the use of video generation models in robotics, highlighting their capabilities in simulating physical interactions, their applications in various learning paradigms, and the challenges faced in deploying them safely and effectively.

Contribution

It provides a comprehensive overview of video models in robotics, discusses current applications, identifies key challenges, and suggests future research directions for safer and more effective integration.

Findings

Video models enable photorealistic, physically consistent simulations.

They serve as expressive world models for complex physical interactions.

Challenges include data costs, hallucinations, and safety concerns.

Abstract

Video generation models have emerged as high-fidelity models of the physical world, capable of synthesizing high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models enable photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. In this survey, we provide a review of video models and their applications as embodied world models in robotics, encompassing cost-effective data generation and action prediction in imitation learning, dynamics and rewards modeling in reinforcement learning, visual planning, and policy evaluation. Further, we highlight important challenges hindering the trustworthy integration of video models in robotics, which include poor instruction following, hallucinations such as violations of physics, and unsafe content generation, in addition to fundamental limitations such as significant data curation, training, and inference costs. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings.