Learning to Generate Diverse Pedestrian Movements from Web Videos with Noisy Labels

TL;DR

This paper introduces PedGen, a generative model trained on a large-scale web video dataset called CityWalkers, capable of producing diverse and realistic pedestrian movements in urban scenes while effectively handling noisy labels and leveraging scene context.

Contribution

The work presents a new dataset CityWalkers and a novel generative model PedGen that filters noisy labels and incorporates 3D scene context for realistic pedestrian movement generation.

Findings

PedGen outperforms baseline methods in pedestrian movement generation.

It achieves zero-shot generalization in real-world and simulated environments.

The approach effectively handles noisy labels and utilizes scene context.

Abstract

Understanding and modeling pedestrian movements in the real world is crucial for applications like motion forecasting and scene simulation. Many factors influence pedestrian movements, such as scene context, individual characteristics, and goals, which are often ignored by the existing human generation methods. Web videos contain natural pedestrian behavior and rich motion context, but annotating them with pre-trained predictors leads to noisy labels. In this work, we propose learning diverse pedestrian movements from web videos. We first curate a large-scale dataset called CityWalkers that captures diverse real-world pedestrian movements in urban scenes. Then, based on CityWalkers, we propose a generative model called PedGen for diverse pedestrian movement generation. PedGen introduces automatic label filtering to remove the low-quality labels and a mask embedding to train with partial labels. It also contains a novel context encoder that lifts the 2D scene context to 3D and can incorporate various context factors in generating realistic pedestrian movements in urban scenes. Experiments show that PedGen outperforms existing baseline methods for pedestrian movement generation by learning from noisy labels and incorporating the context factors. In addition, PedGen achieves zero-shot generalization in both real-world and simulated environments. The code, model, and data will be made publicly available at https://genforce.github.io/PedGen/ .