Non-parametric Memory for Spatio-Temporal Segmentation of Construction Zones for Self-Driving

TL;DR

This paper presents a non-parametric memory system for autonomous vehicles that dynamically remembers, reinforces, and forgets spatio-temporal information to improve real-time segmentation and change detection in complex environments.

Contribution

It introduces a novel non-parametric memory approach that enhances spatio-temporal segmentation by incorporating reinforcement and forgetting mechanisms based on 3D reasoning.

Findings

Effective in remembering past observations of the environment.

Improves detection of changes and occlusions in real-time.

Complementary to static HD maps for dynamic scene understanding.

Abstract

In this paper, we introduce a non-parametric memory representation for spatio-temporal segmentation that captures the local space and time around an autonomous vehicle (AV). Our representation has three important properties: (i) it remembers what it has seen in the past, (ii) it reinforces and (iii) forgets its past beliefs based on new evidence. Reinforcing is important as the first time we see an element we might be uncertain, e.g, if the element is heavily occluded or at range. Forgetting is desirable, as otherwise false positives will make the self driving vehicle behave erratically. Our process is informed by 3D reasoning, as occlusion is key to distinguishing between the desire to forget and to remember. We show how our method can be used as an online component to complement static world representations such as HD maps by detecting and remembering changes that should be superimposed on top of this static view due to such events.