RendBEV: Semantic Novel View Synthesis for Self-Supervised Bird's Eye View Segmentation

TL;DR

RendBEV introduces a self-supervised approach for bird's eye view semantic segmentation using differentiable volumetric rendering, enabling zero-shot performance and improving low-annotation regime results.

Contribution

The paper presents RendBEV, a novel self-supervised training method for BEV segmentation that leverages volumetric rendering and semantic perspective views, reducing reliance on annotated data.

Findings

Achieves competitive zero-shot BEV segmentation results.

Significantly improves performance when fine-tuned with limited labels.

Sets new state-of-the-art results with full labeled data.

Abstract

Bird's Eye View (BEV) semantic maps have recently garnered a lot of attention as a useful representation of the environment to tackle assisted and autonomous driving tasks. However, most of the existing work focuses on the fully supervised setting, training networks on large annotated datasets. In this work, we present RendBEV, a new method for the self-supervised training of BEV semantic segmentation networks, leveraging differentiable volumetric rendering to receive supervision from semantic perspective views computed by a 2D semantic segmentation model. Our method enables zero-shot BEV semantic segmentation, and already delivers competitive results in this challenging setting. When used as pretraining to then fine-tune on labeled BEV ground-truth, our method significantly boosts performance in low-annotation regimes, and sets a new state of the art when fine-tuning on all available labels.