From Single Images to Motion Policies via Video-Generation Environment Representations

TL;DR

This paper introduces VGER, a novel framework that generates environment representations from a single RGB image using video generation models, enabling collision-free motion planning for robots.

Contribution

The paper presents VGER, a new method that leverages video generation and 3D modeling to create environment representations from a single image for motion planning.

Findings

VGER produces smooth, geometry-aware robot motions from a single image.

The framework effectively handles diverse indoor and outdoor environments.

VGER outperforms existing monocular depth-based methods in motion generation.

Abstract

Autonomous robots typically need to construct representations of their surroundings and adapt their motions to the geometry of their environment. Here, we tackle the problem of constructing a policy model for collision-free motion generation, consistent with the environment, from a single input RGB image. Extracting 3D structures from a single image often involves monocular depth estimation. Developments in depth estimation have given rise to large pre-trained models such as DepthAnything. However, using outputs of these models for downstream motion generation is challenging due to frustum-shaped errors that arise. Instead, we propose a framework known as Video-Generation Environment Representation (VGER), which leverages the advances of large-scale video generation models to generate a moving camera video conditioned on the input image. Frames of this video, which form a multiview dataset, are then input into a pre-trained 3D foundation model to produce a dense point cloud. We then introduce a multi-scale noise approach to train an implicit representation of the environment structure and build a motion generation model that complies with the geometry of the representation. We extensively evaluate VGER over a diverse set of indoor and outdoor environments. We demonstrate its ability to produce smooth motions that account for the captured geometry of a scene, all from a single RGB input image.