Parametric Depth Based Feature Representation Learning for Object Detection and Segmentation in Bird's Eye View

TL;DR

This paper introduces a parametric depth-based feature transformation method for bird's-eye-view perception in autonomous driving, improving object detection and segmentation by leveraging geometry and visibility information.

Contribution

We propose a novel parametric depth distribution modeling approach for BEV feature transformation, addressing memory issues and hallucination problems in perception tasks.

Findings

Outperforms existing methods on nuScenes dataset for detection and segmentation.

Provides reliable visibility-aware estimations to reduce hallucination.

Enhances downstream perception tasks with geometry-informed features.

Abstract

Recent vision-only perception models for autonomous driving achieved promising results by encoding multi-view image features into Bird's-Eye-View (BEV) space. A critical step and the main bottleneck of these methods is transforming image features into the BEV coordinate frame. This paper focuses on leveraging geometry information, such as depth, to model such feature transformation. Existing works rely on non-parametric depth distribution modeling leading to significant memory consumption, or ignore the geometry information to address this problem. In contrast, we propose to use parametric depth distribution modeling for feature transformation. We first lift the 2D image features to the 3D space defined for the ego vehicle via a predicted parametric depth distribution for each pixel in each view. Then, we aggregate the 3D feature volume based on the 3D space occupancy derived from depth to the BEV frame. Finally, we use the transformed features for downstream tasks such as object detection and semantic segmentation. Existing semantic segmentation methods do also suffer from an hallucination problem as they do not take visibility information into account. This hallucination can be particularly problematic for subsequent modules such as control and planning. To mitigate the issue, our method provides depth uncertainty and reliable visibility-aware estimations. We further leverage our parametric depth modeling to present a novel visibility-aware evaluation metric that, when taken into account, can mitigate the hallucination problem. Extensive experiments on object detection and semantic segmentation on the nuScenes datasets demonstrate that our method outperforms existing methods on both tasks.