Monocular Plan View Networks for Autonomous Driving

TL;DR

This paper introduces a monocular plan view network for autonomous driving that transforms camera observations into a bird's eye view, improving environment understanding and control accuracy in simulated environments.

Contribution

It proposes a novel plan view transformation for monocular images, enabling better reasoning about distances and object positions for autonomous driving.

Findings

Reduces collisions by over 50% compared to detection-based methods.

Achieves an order of magnitude fewer collisions than pixel-based policies.

Demonstrates effectiveness in a realistic simulation environment.

Abstract

Convolutions on monocular dash cam videos capture spatial invariances in the image plane but do not explicitly reason about distances and depth. We propose a simple transformation of observations into a bird's eye view, also known as plan view, for end-to-end control. We detect vehicles and pedestrians in the first person view and project them into an overhead plan view. This representation provides an abstraction of the environment from which a deep network can easily deduce the positions and directions of entities. Additionally, the plan view enables us to leverage advances in 3D object detection in conjunction with deep policy learning. We evaluate our monocular plan view network on the photo-realistic Grand Theft Auto V simulator. A network using both a plan view and front view causes less than half as many collisions as previous detection-based methods and an order of magnitude fewer collisions than pure pixel-based policies.

Equations

$$(d,{\theta }_l,l,w,h{)}_k=E_{3D}(f_k)\text{ for object }k$$

$$\theta =({\theta }_l-\arctan \frac{x}{f})\mathrm{mod}2\pi .$$

$$X=\frac{dx}{f},Y=d.$$

$$o_{PV}=\left[\begin{array}{c}X\\ Y\end{array}\right]+R(\theta )\left[\begin{array}{c}\pm \frac{w}{2}\\ \pm \frac{l}{2}\end{array}\right]$$

Full text

I Introduction

Autonomous driving promises accessibility, convenience, and safety on the road. Some recent approaches for training driving policies use demonstration data from monocular RGB images [1, 2, 3]. However, the large observation space of monocular RGB images may introduce many correlations that do not hold over in an on-policy evaluation [4]. Moreover, a convolutional network assumes the wrong invariances when used on a first-person image. For example, a ConvNet is invariant to image plane translation but not to perspective or rotation. While these invariances are useful for classification tasks, relative and absolute positions matter in driving. A tree on the side of the road can be safely ignored, while a tree in the path of the vehicle quickly leads to an accident. Perspective effects in an image ensure that a plain monocular ConvNet needs to learn different filters for each object in each position and distance from the car.

The autonomous driving task requires reasoning about free space and other drivers’ future positions. Classical navigation methods tend to use range-finders to draw a map of free space, then compute controls from the top-down map. As shown by Tamar et al. , ConvNets can perform value iteration and planning on top-down navigation environments [5]. To avoid relying entirely on LiDAR sensors, which are expensive and may misread rain or transparent surfaces, many modern approaches to deep learning for autonomous driving use first-person RGB views of the road. These are naturally available from cameras on a car but which are a less well structured representation of the state.

While a first-person view image provides information about object positions and depths, the free space and overall structure of the scene is implicit rather than explicit. Hallucinating a top-down view of the road makes it easier to learn to drive as free and occupied spaces are explicitly represented at a constant resolution through the image. A top-down view also can include the agent into the observation.

Intuitively, two- and three-dimensional object detectors [6] should provide useful information for autonomous driving. However, despite the advances in 3D object detection [7, 8, 9], few attempts have been made to use these detections in conjunction with deep learning in robotics. Prior methods included them either in the monocular view or with an attention based pooling [10, 11]. Both approaches struggle to reason about the intricate three dimensional interplay between cars in complex traffic patterns. Our approach leverages the spatial reasoning power of ConvNets to analyze and react to these traffic patterns.

In this paper, we propose Monocular Plan View Networks (MPV-Nets) to combine the advantages of first-person views, which are naturally available from the driving agent, and top down views, which enable reasoning about objects and space. The MPV-Nets require only a monocular first-person image and infers the depths and poses of the objects within the scene. With this 3D information, the MPV-Nets reproject the objects into an overhead plan view.

This top-down image allows the MPV-Nets policy to easily extract and use features related to spatial positions of objects in the scene. A two dimensional ConvNet learns to predict an action from this overhead view. This low-dimensional representation emphasizes free-space around the agent in contrast to first-person images, where weather or style variations may confuse the policy.

Our entire pipeline is end-to-end trained to imitate expert trajectories with intermediate detection losses. We use detection and 3D estimation supervision for each object during training and infer these during testing. We evaluate our driving policy on the photo-realistic Grand Theft Auto V simulator. Our experiments show that an MPV policy performs significantly better than standard monocular policies with and without objects detections. The MPV policy both travels farther and experiences fewer collisions and interventions than front-view policies. Our model also outperforms using only the plan view, indicating that the front view image pixels do provide necessary information.

II Related Work

Our inferred monocular plan view representation has some similarities with occupancy grids generated by LiDAR sensors: we contrast free space with space occupied by vehicles and pedestrians. Recent works have used convolutional networks with LiDAR data for localization [12], scene flow estimation [13], 3D detection and tracking [14, 15, 16, 17, 18, 19, 20], intention prediction [21], and semantic segmentation [22]. In particular, Luo et al. propose a joint framework for 3D detection, tracking, and motion forecasting via 3D convolution over a bird’s eye view representation of 3D world [15]. Yang et al. propose a map-aware single-stage 3D detector on the LiDAR bird’s eye view [19]. Petrovskaya et al. use the concept of a “virtual scan”, which is a 2D representation of the 3D rangefinder output and which accounts for the ego-motion of the car [23]. Darrell et al. construct a plan view model from multiple cameras to track objects from a stereoscopic 3D estimate in order to help track entities and obtain a globally consistent reference frame [24].

Our work, on the other hand, uses the plan view as new input modality to a deep network. Our work differs as we only use an RGB camera sensor and specifically detect vehicles and pedestrians in order to construct the view. Palazzi et al. generate top-down vehicle coordinates from a first-person RGB input but require paired top and first-person images and do not use these in a driving policy [25]. Our method only re-projects detected objects and can be trained from monocular video and three dimensional location or from tracking information.

Training neural network policies for driving has a long history. ALVINN [26] was the first usage of neural network for end-to-end driving from image data. Muller et al. introduce a ConvNet-based obstacle avoidance system for off-road mobile robots [27]. More recently, Bojarski et al. [1, 28] and Xu et al. [2] revisited end-to-end imitation learning, training a policy to output steering commands directly form RGB images. Muller et al. added a bias to the deep policies by training a policy on the semantic segmentation of the scene as well as the image [29]. The segmentation module enabled better transfer from synthetic to real, but does not include information about the 3D layout of the scene.

Another form of domain knowledge was used by Chen et al. [30] and Sauer et al. [31], who chose a number of “affordances” to predict and supervise, such as the distance between cars and lanes, or the status of the streetlights. It is clear that adding some domain knowledge or model bias can significantly improve driving performance and transfer. However, supervising specific affordances can be an ad-hoc process. Our approach instead provides 3D information about the scene in a representation that showcases spatial relations between objects.

III Monocular Plan View Networks

Figure 2 shows an overview of our proposed MPV-Net. Given a monocular first person view RGB image, a 2D detector finds the image coordinates of all objects in the scene. The 3D estimation network then computes the depth, rotation, and 3D size of each object. A reprojection layer renders each 3D object into a plan view image using the 3D bounding box information. The original first-person image and the generated plan view are each passed through separate convolutional networks and globally pooled into two feature vectors. The resulting features are concatenated and fed to a fully connected layer which outputs a discrete action. This pipeline is summarized in Algorithm 1.

2D detection

Our goal is to construct and utilize a plan view for policy learning. However, our observation space is a simple first-person view image $I_{FV}$ from a dash camera of a car. In order to construct the plan view we first detect all cars and pedestrians in an image. We use an off-the-shelf Mask RCNN [32] detector to extract a 2D bounding box $(u,v)_{k}$ for each object $k$. Next, we estimate the 3D position, size, and heading of each object.

3D estimation

The 3D module estimates the object’s 3D pose: plan view 2D position $(X,Y)$, angle in plane (yaw) $\theta$, and size $(l,w,h)$. As input, the 3D module takes a bounding box $(u,v)_{i}$ and the front view image $I_{FV}$. Inspired by Mousavian et al. [6], we train a network $E_{3D}$ with three sub-networks to estimate the depth $d$, local orientation $\theta_{l}$, and size $(l,w,h)$ of each object. Figure 3 shows an overview of the module. All three sub-networks share a common fully convolutional base network. The base network extracts dense features from the original input image. For each bounding box, we extract a 3D feature $f_{k}$ from the base network using ROIAlign [32]. The three sub-networks then predict their corresponding outputs from these ROIAlign features:

[TABLE]

In addition, we compute the relative horizontal position $x_{k}$ from the center of the 2D bounding box.

For a particular object, we compute the global rotation $\theta$ in the camera coordinate system from $\theta_{l}$, as shown in Figure 3. Given the horizontal distance from the vertical center of image $x$ and the camera’s focal length $f$,

[TABLE]

The relative plan view location of the object is given by

[TABLE]

The width $w$ and length $l$ of an object are directly estimated in plan-view space. We obtain the ground truth focal length $f$ directly from the simulator and learn all network parameters with direct supervision.

Our 3D estimation network $E_{3D}$ does not share a base network with our detector for two reasons: First, it significantly simplifies training. Second, the two networks learn drastically different features. For example, a detection network should be scale invariant, which works well for size and rotation estimates, but provides little information about the depth of an object. However, in order to perform good monocular depth estimation, the base network needs to detect the absolute 2D size of objects and store it in an ROI feature. Only a prior on the size of objects will yield a good depth estimate. A good depth estimation network also heavily exploits priors over object locations, which a detector should ignore.

Reprojection

The reprojection layer converts the 3D estimates ($\theta$, $X$, $Y$, $l$, $w$) of all objects into a plan view image. We create a separate plan view image for each class of objects. The plan view image is assigned a value $1$ if any object occupies the spatial location corresponding to a pixel and [math] otherwise. For a particular object centered at $X$, $Y$ with size $l,w,h$ and rotation $\theta$, the reprojection layer computes the extent of the object in the plan view. We describe each object in plan view as a rotated bounding box with four corners defined as

[TABLE]

where $R(\theta)$ is the rotation matrix for angle $\theta$.

There are several options to choose from when deciding how to structure the reprojection, such as what information to include in the view and what orientation to face. In this work, we focus on reprojecting vehicles and pedestrians, but one could also overlay a map if available, or detect street signs, lights, lanes, etc. Here we explore two possible orientations of the plan view: First, the scene is rendered relative to the agent with the agent always facing upwards as in Figure 1. We call this type of MPV the direction-of-travel view. This may lead to better generalization because driving south would look the same as driving north. However, this view is easily distracted by the agent’s own actions and is not invariant to ego-motion and rotation. The second option we explore uses a constant frame of reference by always placing north at the top. We name this north-up view.

Once we fix the reference frame of the plan view, we render each moving object (vehicle or pedestrian) into a top-down image. We use an aliased rendering technique and set all pixels inside an object box to $1$. This step is not differentiable but could easily be replaced with a differentiable anti-aliased alternative [33] if an end-to-end gradient is required. Figure 3 shows the complete 3D module.

Policy Network

Given the generated MPV image, we use a convolutional network to extract visual and spatial features for both RGB image and MPV images. The visual ConvNet $C_{FV}$ is pre-trained on ImageNet [34], while the spatial ConvNet $C_{PV}$ is trained from scratch. The concatenation of both visual and spatial features is fed into a linear policy $\pi$ to produce actions. Figure 4 shows the policy network structure. We train the the ConvNets and $\pi$ jointly with imitation learning.

IV Experiments

We evaluate MPV-Nets in a realistic simulated driving environment. All models are trained with imitation learning, and their driving performance is evaluated through several metrics, including the number of collisions and the distance traveled. The aim of the experiments is to determine whether explicit use of plan view does help learn driving policies as well which representation of the 3D object is best. As the 3D detections are inferred from the same front view image used by the baseline models, no additional information is generated at test time by the detections. However, as seen in Figure 8, we find that this structured representation does improve final driving performance. We also show in Table II that the 3D detections alone are not sufficient for driving: a policy trained solely on the MPV without the front view image features is unable to predict the expert’s actions.

IV-A Experimental Settings

Dataset

All models (baselines, related works, and our MPV-Nets) are trained on a dataset of (image, action) pairs generated from Grand Theft Auto V. Leveraging the in-game navigation system as the expert policy, we collect 2.5 million training frames over $1000$ random paths over $2$km at 12 fps. With the default cloudy weather, we collect driving demonstrations from 8:00 am to 7:00 pm. The horizontal field of view (FoV) for the dash cam is fixed as fixed $60^{\circ}$. Inspired by DAgger [35], we randomly add noise to the expert’s action every 30s to augment the data with examples of error recovery. The noisy control actions along the following seven frames are removed when saving the data for training to avoid imitating the noisy behavior.

Each frame includes a front-view RGB image and control signals such as speed, angle, throttle, steering, and brake. We discretize the target control into 9 actions: (left, straight, right) $\times$ (fast, slow, stop). The 3D estimation module is also trained on the simulated data: we extract 3D ground-truth pose labels for each object relative to the agent. These are not used during evaluation unless the experiment is labeled “ground truth”.

Training the models with imitation learning

All models are trained on the dataset to predict the expert’s action at each frame using a cross-entropy loss on the discretized action space. For the models using 3D detection (MPV and FVD), the 3D detection module is first trained to predict the relative position, depth, and size of cars and pedestrians in the image. Then, the 3D detection is frozen, and the convolutional networks that extract global features from the front view image and re-projected image, as well as the policy layer, are trained on the behavioral cloning objective.

Off-policy Evaluation

As an initial evaluation, we can compare the perplexities of different models. The perplexity, which calculates the negative log likelihood of the expert data under the distribution predicted by the model (lower is better), measures how well the model fits the data. Lower perplexity does not always indicate high on-policy performance, but high perplexity (as in Table II, bottom rows) indicates that the model has underfit.

On-policy Evaluation

We evaluate the models in $8$ locations unseen during training: $2$ highway and $6$ urban intersections. Figure 5 demonstrates some example scene layouts in our simulation environment. For each test location, we run the model for 10 independent roll-outs lasting around $800$ steps, 10 minutes of game time, each. Because the model predicts discretized actions, a hand-tuned PID controller (shared for all models) is used to convert the discrete actions into continuous control signals. Collisions are counted as detected by the environment. If the agent gets stuck or stops moving for $30$ seconds, we let the in-game AI intervene and count the intervention. Not all collisions lead to interventions. The time and distance traveled by the expert during an intervention are not counted in the evaluation.

The models are evaluated with several metrics. For each roll-out, we count the number of collisions, the number of interventions, and the distances traveled. To aggregate across roll-outs, we compute the average distance driven between AI interventions, the number of collisions per 100m, and the number of interventions per 100m.

IV-B Implementation

We implement our system with the open-source deep learning framework PyTorch [36]. We extend the work of Krähenbühl [37] and extract ground truth object 3D pose ground truth from Grand Theft Auto V. In order to provide a fair evaluation, all models predict 1 action every 7 frames, and the game is paused during the model’s computation. In this way, all the models are evaluated on the same number of forward passes.

For the 2D detection module, we use a COCO [38] pre-trained Mask RCNN [32] based on Detectron [39]. We use a 34-layer DLA model [40] as the backbone architecture for 3D estimation and policy network. The design of the 3D sub-network is the stack of three $3\times 3$ convolutional layers.

We train our policy network with Adam [41] for 2 epochs. The initial learning rate is $0.001$ and the batch-size is $64$. The original input RGB image is $1920\times 1080$. We resize the given image into $640\times 352$ for both training and testing. The size of the MPV image is $512\times 512$, spatially covering 64m$\times$64m of the environment. We ignore the objects whose depth is greater than 64m for all experiments. We only keep the objects within 32m to the right and left. We do not use any visual data augmentation during training.

IV-C Evaluation of 3D Esimation

We evaluate the 3D estimation module for pedestrian and vehicle in Table I. We utilize the depth and orientation evaluation metrics from Eigen et al.  [42] and KITTI [43]. The depth estimation is evaluated by both error and accuracy metrics, while rotation is evaluated by orientation score (OS). Size prediction is measured by the accuracy of the object volume estimation : $Dim=\min(V_{\mathtt{pred}}/V_{\mathtt{gt}},V_{\mathtt{gt}}V_{\mathtt{pred}})$ with an upper bound $1$, where $V=lwh$ is the volume of a 3D box.

Figure 5 shows a comparison between predicted MPV images and the ground truth ones on randomly picked images. The 3D module is robust to various scenarios under different weather and timing, but is not perfect. Common errors include missing detections or wrong orientations.. However, the imperfect perception system does not significantly impact the driving policy. The policy predicts an almost identical distribution over actions for either predicted or ground truth MPV images. The MPV policy network is robust to an imperfect perception module.

V Results

We evaluate all models on their on-policy driving performance. For each model we compute the average distance between two interventions, the number of interventions per $100m$, and the number of collisions per $100m$. The best models should drive long distances between interventions and have few interventions and collisions.

Design Choice Ablation

We evaluate the design choices available for the plan view reprojection: (1) Rendering the plan view in direction of travel (Travel) of the agent or in a fixed north-up direction (North). (2) The plan view can include the ego position history of the agent within the image. (3) The plan view can include a coarse road map of the environment using the GPS position of the agent. See Figure 6 for a visual example.

Table II shows the results in terms of off-policy perplexity, cross entropy of predicted actions on a held-out validation set, and our on-policy metrics. The best models use neither a map nor the agent’s ego motion. The orientation of the MPV is not important. Including the agent’s history performs much better in the off-policy perplexity evaluation but fails at on-policy driving. This is due to the learning algorithm used: imitation learning, and specifically behavioral cloning. It is prone to overfitting to the expert’s data distribution, as supported by [27]. In contrast, the models trained on only the MPV without access to the front view image never learn to model the dataset. We were unable to test the on-policy performance of these models.

For all remaining experiments we use a direction-of-travel oriented plan view without a map or history.

MPV-Net evaluation

We compare our model against several baselines, prior methods, and ablations. The baseline method is based on the the network by Xu et al.  [2], which uses a single convolutional network policy trained on the first-person view. The attention method extends the baseline with a pixel-level attention mechanism, learned end-to-end with the policy. This is similar to Kim et al.  [44]. Attention learns a weight mask for each feature location in the first-person view and MPV feature extractor. This attention then guides a weighted average pooling. The object method builds on Wang et al.  [10], and combines 2D object detections and deep image features, but lacks explicit depth supervision. We augment all our baselines with an additional front view depth (FVD) estimate without a plan view in the baseline+FVD and attention+FVD experiment. See Figure 6 for a visual example of the FVD.

Figure 8 shows the main comparison on the on-policy evaluation. A model that uses explicit depth information through either the MPV image or as an RGB-D image always improves performance when compared to the same model without depth. Unsurprisingly, the simplest models (baseline and attention), which use no supervision other than the expert’s actions, perform significantly worse than all other models. This even holds for predicted depth estimates and detections.

We also find that, although the FVD and MPV are supervised with similar information, the MPV models experience fewer interventions and collisions, especially in the busy urban environments. This supports our hypothesis that MPV provides a better representation for policy learning.

To better understand each model’s performance, we additionally plot the portion of interventions that are due to collisions in Figure 7. We see that while the attention+MPV model has the fewest interventions, both MPV-based models have the lowest proportion of interventions caused by collisions in the urban setting. This indicates that the plan view, more-so than simple depth estimation, is helpful for navigating through intersections.

Figure 9 shows example rollouts of the attention+MPV-Net policy using the learned 3D estimator. In the top row, the plan view shows the pedestrian clearly in the way of the car, causing the car to brake appropriately. As we use the direction-of-travel reprojection, the detected entities rotate through the image when the car is turning, as shown in the second row of the figure.

VI Discussion

Monocular Plan View Networks (MPV-Nets) provide a top-down view of the driving environment using a standard front-view image and 3D object localization. Our results show that this representation improves driving performance in the GTA-V simulator, even when 3D detections are imperfect. The quality of the driving policy does not degrade even when using noisy predicted 3D pose estimates. By combining our MPV-Net with a pixel-level attention, we achieve almost perfect performance (0 collisions or interventions) on the highway driving task even when using predicted 3D detections. City intersections remain a challenge, but our plan view network provides promising results. In comparison to traditional off-policy dataset evaluations, our on-policy evaluation provides a realistic understanding of how good a policy can be when trained from demonstrations.

MPV-Nets provide an intuitive way of incorporating trajectory history, maps, and object attention. While our ablations found that these additions hurt performance in the behavioral cloning setup, we expect that they would be very useful for planning or reinforcement learning based approaches. The plan view can intuitively and cleanly represent nearby vehicles’ trajectories in a way that is accessible to a deep convolutional network.