GHOST: Ground-projected Hypotheses from Observed Structure-from-Motion Trajectories

TL;DR

This paper introduces a scalable self-supervised method for segmenting vehicle trajectories from monocular images using structure-from-motion, enabling autonomous navigation without manual annotations in complex urban environments.

Contribution

It presents a novel approach that leverages ego-vehicle motion as implicit supervision to generate training labels for trajectory segmentation from monocular videos.

Findings

Achieves reliable trajectory prediction on NuScenes dataset.

Successfully transfers to electric scooter platform with minimal fine-tuning.

Implicit scene understanding enables generalizable path proposals.

Abstract

We present a scalable self-supervised approach for segmenting feasible vehicle trajectories from monocular images for autonomous driving in complex urban environments. Leveraging large-scale dashcam videos, we treat recorded ego-vehicle motion as implicit supervision and recover camera trajectories via monocular structure-from-motion, projecting them onto the ground plane to generate spatial masks of traversed regions without manual annotation. These automatically generated labels are used to train a deep segmentation network that predicts motion-conditioned path proposals from a single RGB image at run time, without explicit modeling of road or lane markings. Trained on diverse, unconstrained internet data, the model implicitly captures scene layout, lane topology, and intersection structure, and generalizes across varying camera configurations. We evaluate our approach on NuScenes, demonstrating reliable trajectory prediction, and further show transfer to an electric scooter platform through light fine-tuning. Our results indicate that large-scale ego-motion distillation yields structured and generalizable path proposals beyond the demonstrated trajectory, enabling trajectory hypothesis estimation via image segmentation.