Estimation of Appearance and Occupancy Information in Birds Eye View from Surround Monocular Images

TL;DR

This paper introduces a novel method to generate a comprehensive Birds-eye View (BEV) that includes appearance and occupancy details of traffic participants from multiple monocular cameras, enhancing scene understanding for autonomous driving.

Contribution

It proposes a new approach to produce a detailed BEV with appearance information using learned image embeddings from surround monocular cameras, improving scene interpretation.

Findings

Effective scene representation with appearance and occupancy details

Improved downstream task performance such as object tracking

Validated on synthetic CARLA dataset

Abstract

Autonomous driving requires efficient reasoning about the location and appearance of the different agents in the scene, which aids in downstream tasks such as object detection, object tracking, and path planning. The past few years have witnessed a surge in approaches that combine the different taskbased modules of the classic self-driving stack into an End-toEnd(E2E) trainable learning system. These approaches replace perception, prediction, and sensor fusion modules with a single contiguous module with shared latent space embedding, from which one extracts a human-interpretable representation of the scene. One of the most popular representations is the Birds-eye View (BEV), which expresses the location of different traffic participants in the ego vehicle frame from a top-down view. However, a BEV does not capture the chromatic appearance information of the participants. To overcome this limitation, we propose a novel representation that captures various traffic participants appearance and occupancy information from an array of monocular cameras covering 360 deg field of view (FOV). We use a learned image embedding of all camera images to generate a BEV of the scene at any instant that captures both appearance and occupancy of the scene, which can aid in downstream tasks such as object tracking and executing language-based commands. We test the efficacy of our approach on synthetic dataset generated from CARLA. The code, data set, and results can be found at https://rebrand.ly/APP OCC-results.