Streetscapes: Large-scale Consistent Street View Generation Using Autoregressive Video Diffusion

TL;DR

This paper introduces Streetscapes, a scalable autoregressive video diffusion method for generating long, city-scale street view sequences conditioned on language and map data, maintaining high visual consistency.

Contribution

The authors develop a novel autoregressive video diffusion framework with a temporal imputation technique, enabling long-range, city-scale street view synthesis conditioned on language and map inputs.

Findings

Can generate city-scale street views spanning several blocks.

Maintains visual quality and consistency over long sequences.

Effective conditioning on language and map data for controllable generation.

Abstract

We present a method for generating Streetscapes-long sequences of views through an on-the-fly synthesized city-scale scene. Our generation is conditioned by language input (e.g., city name, weather), as well as an underlying map/layout hosting the desired trajectory. Compared to recent models for video generation or 3D view synthesis, our method can scale to much longer-range camera trajectories, spanning several city blocks, while maintaining visual quality and consistency. To achieve this goal, we build on recent work on video diffusion, used within an autoregressive framework that can easily scale to long sequences. In particular, we introduce a new temporal imputation method that prevents our autoregressive approach from drifting from the distribution of realistic city imagery. We train our Streetscapes system on a compelling source of data-posed imagery from Google Street View, along with contextual map data-which allows users to generate city views conditioned on any desired city layout, with controllable camera poses. Please see more results at our project page at https://boyangdeng.com/streetscapes.