Future Urban Scenes Generation Through Vehicles Synthesis

TL;DR

This paper introduces a two-stage deep learning method for predicting future urban scene appearances by modeling individual vehicles with novel view synthesis, enabling diverse and realistic scene generation conditioned on input trajectories.

Contribution

It presents a novel per-object view synthesis approach for future scene prediction, incorporating interpretable information and enabling multi-modal, conditioned scene generation.

Findings

Outperforms traditional end-to-end methods on CityFlow dataset

Generates diverse realistic scene futures

Models each actor independently for interpretability

Abstract

In this work we propose a deep learning pipeline to predict the visual future appearance of an urban scene. Despite recent advances, generating the entire scene in an end-to-end fashion is still far from being achieved. Instead, here we follow a two stages approach, where interpretable information is included in the loop and each actor is modelled independently. We leverage a per-object novel view synthesis paradigm; i.e. generating a synthetic representation of an object undergoing a geometrical roto-translation in the 3D space. Our model can be easily conditioned with constraints (e.g. input trajectories) provided by state-of-the-art tracking methods or by the user itself. This allows us to generate a set of diverse realistic futures starting from the same input in a multi-modal fashion. We visually and quantitatively show the superiority of this approach over traditional end-to-end scene-generation methods on CityFlow, a challenging real world dataset.