Dynamic Occupancy Grid Prediction for Urban Autonomous Driving: A Deep Learning Approach with Fully Automatic Labeling

TL;DR

This paper presents a deep learning approach for long-term dynamic occupancy grid prediction in urban autonomous driving, utilizing automatic labeling and Bayesian filtering to improve environment understanding and interaction modeling.

Contribution

It introduces a fully automatic labeling method combined with a deep CNN for long-term prediction of complex urban scenarios, addressing sensor data imbalance and interaction modeling.

Findings

The method achieves accurate long-term predictions in urban environments.

Automatic label generation enables unsupervised training.

The approach outperforms Monte-Carlo simulation in evaluations.

Abstract

Long-term situation prediction plays a crucial role in the development of intelligent vehicles. A major challenge still to overcome is the prediction of complex downtown scenarios with multiple road users, e.g., pedestrians, bikes, and motor vehicles, interacting with each other. This contribution tackles this challenge by combining a Bayesian filtering technique for environment representation, and machine learning as long-term predictor. More specifically, a dynamic occupancy grid map is utilized as input to a deep convolutional neural network. This yields the advantage of using spatially distributed velocity estimates from a single time step for prediction, rather than a raw data sequence, alleviating common problems dealing with input time series of multiple sensors. Furthermore, convolutional neural networks have the inherent characteristic of using context information, enabling the implicit modeling of road user interaction. Pixel-wise balancing is applied in the loss function counteracting the extreme imbalance between static and dynamic cells. One of the major advantages is the unsupervised learning character due to fully automatic label generation. The presented algorithm is trained and evaluated on multiple hours of recorded sensor data and compared to Monte-Carlo simulation.

Equations

$$P_{\mathrm{O}}=0.5\cdot M_{\mathrm{O}}+0.5\cdot (1-M_{\mathrm{F}})$$

$$\left(P_{\mathrm{O}}(k)\right)=\left(P_{\mathrm{O}}(1),P_{\mathrm{O}}(2),P_{\mathrm{O}}(3),...\right)$$

$$\begin{array}{rl}\left(P_{\mathrm{O}}(k)\right)& =P_{\mathrm{O},\mathrm{s}}+\left(P_{\mathrm{O},\mathrm{d}}(1),P_{\mathrm{O},\mathrm{d}}(2),P_{\mathrm{O},\mathrm{d}}(3),...\right)\\ & =P_{\mathrm{O},\mathrm{s}}+\left(P_{\mathrm{O},\mathrm{d}}(k)\right)\end{array}$$

$$L=L_{\mathrm{s}}+L_{\mathrm{d}}$$

$$L_{\mathrm{s}}=\frac{1}{2}{\lambda }_{\mathrm{s}}\sum _{c=1}^{W\times H}{\left(P_{\mathrm{O},\mathrm{s}}(c)-{\hat{P}}_{\mathrm{O},\mathrm{s}}(c)\right)}^2$$

$${\lambda }_{c,k}=|1+{\lambda }_k{P}_{\mathrm{O},\mathrm{d}}(c,k)|$$

$$L_{\mathrm{d}}=\frac{1}{2}\sum _{k=1}^{|\mathcal{T}|-1}\sum _{c=1}^{W\times H}{\lambda }_{c,k}{\left(P_{\mathrm{O},\mathrm{d}}(c,k)-{\hat{P}}_{\mathrm{O},\mathrm{d}}(c,k)\right)}^2$$

$$\frac{\delta L_{\mathrm{d}}}{\delta {\hat{P}}_{\mathrm{O},\mathrm{d}}}=-|1+{\lambda }_k{P}_{\mathrm{O},\mathrm{d}}(c,k)|\left(P_{\mathrm{O},\mathrm{d}}(c,k)-{\hat{P}}_{\mathrm{O},\mathrm{d}}(c,k)\right).$$

Full text

I Introduction

Research of the past decades gained great success in the field of autonomous driving. Comparing automated vehicles to human drivers, humans can compensate their comparably long reaction time with the ability to estimate a future scene evolution. This also enables strategic and forward-looking behavior, hard to compensate with the fast reaction and precise perception of machines. Long-term situation prediction for automated driving is therefore a major challenge still to overcome. We tackle a highly complex urban environment where cars, trucks, bikes and pedestrians share the road. Road users react individually, but have type-specific motion constraints. Most important, humans, walking or driving, do not act independently in such scenarios, but rather interact and account for possible behavior of others. Consequently, an individual decision and path may have consequences to the future behavior of someone or everyone else. Such traffic scenarios might, hence, evolve into many possible future scene constellations, motivating the need for a probabilistic representation of the prediction result. In short, long-term situation prediction for urban autonomous driving should be able to incorporate interactions of all known traffic participants and, due to the uncertain nature of the problem, handle the variance of possible outcomes.

Models of interactions and motion dynamics are common tools to improve prediction. Literature covers long-term prediction using engineered and learned models. Manually designed dependencies in road user prediction, e.g., using a digital map or the Intelligent Driver Model [1], usually underlie quite limiting assumptions, e.g., a lane following [2] or car following [3] scenario. In addition, hand crafted models considering context and object relations tend to be high dimensional [1, 4], which motivates machine learning as an alternative. Concordant to the great success of machine learning for classification, methods were proposed to predict a future maneuver with regard to discrete maneuver classes, e.g., stop, go-straight or turn [5, 6]. However, predicting a trajectory in terms of a time series differs from predicting a maneuver class in terms of generalization. To gain a spatio-temporal distribution at the algorithm output, it is, however, a matter of data representation. Wiest [7], e.g., predicted parameters of Chebyshev polynomials considered as random variables in a Gaussian mixture model.

In contrast to the algorithm output, considering time series of raw sensor data at the input, several technical problems arise despite data representation. Varying sample frequency, asynchronous sensors, scaling and translation in time domain, or out of sequence input are examples which are challenging for machine learning [8], but well-studied by the sensor fusion community [9, 10]. From a machine learning perspective, instead of learning temporal features from time series and facing the aforementioned problems, state variables of a single time stamp are related by first-order differential equations representing short-term dynamic features. In addition, we assume, that a lack of long-term input sequences can be compensated by spatial context. This assumption is inspired by research in stock market prediction [11] where it is known that using the past time series is not sufficient to predict the future stock trend [12]. Moreover, it is not clear if they are helpful at all [13]. However, prediction performance can be improved [11] by context information like twitter moods [14] or online chats [15].

The spatial context can be seen in a dynamic occupancy grid map (DOGMa), as illustrated in Fig. 1. The DOGMa uses Bayesian filtering [16] to fuse a variety of sensors and represents the entire environment, static and dynamic, in a bird’s eye view. Each multichannel pixel, or cell, contains probabilistic occupancy and velocity estimates. Moreover, the image-like data structure of DOGMas naturally suggests using convolutional neural networks (CNNs) to exploit contextual information and dependencies between different objects and the given infrastructure.

Therefore, in this work, the strengths of engineered and learning based methods are combined. A convolutional neural network (CNN) is used to model long-term motion, and a Bayesian estimate of the current dynamic environment is used at the input. For training, no hand crafted labeling has to be performed, however, an algorithm separating rare dynamic from static grid cells was applied using future and past estimates. First of all, this enables balancing between static and dynamic regions during training, but moreover a learned segmentation of the current environment outperforming engineered approaches.

The remaining paper is organized as follows. In Section II, we give an overview of dynamic occupancy grid mapping. Section III explains fully automatic label generation for unsupervised learning. IV gives a short overview of the employed CNN architecture. In Section V, a pixel balancing loss function is introduced, counteracting the high underrepresentation of dynamic cells. Results are shown in Section VII followed by conclusions in VIII.

II Filtered Dynamic Input

A dynamic occupancy grid map (DOGMa) [16], fed by multiple sensors, can provide a bird’s eye image like $360\text{\,}\mathrm{\SIUnitSymbolDegree}$ representation of the environment, as illustrated in Fig. 1. The DOGMa data is provided in $\mathbb{R}^{|\Omega|\times W\times H}$ with the spatial width $W$ and height $H$ pointing in east and north direction, respectively. The channels $\Omega=\left\{M_{\mathrm{O}},M_{\mathrm{F}},v_{\mathrm{E}},v_{\mathrm{N}},\sigma^{2}_{v_{\mathrm{E}}},\sigma^{2}_{v_{\mathrm{N}}},\sigma^{2}_{v_{\mathrm{E}},v_{\mathrm{N}}}\right\}$ contain the Dempster-Shafer masses for free space $M_{\mathrm{F}}\in\left[0,1\right]$ and occupancy $M_{\mathrm{O}}\in\left[0,1\right]$, the velocity pointing east $v_{\mathrm{E}}$ and north $v_{\mathrm{N}}$, as well as the velocity variances and covariance. The occupancy probability is calculated by

[TABLE]

where a high $P_{\mathrm{O}}$ refers to a dark pixel in Fig. 1. All channels $\Omega$ and extensions can be used to feed a neural net, although Fig. 1 only contains $3$ RGB-channels for illustration.

Feeding raw sensor data into the CNN is intentionally avoided, due to two main advantages:

1. DOGMa cells provide a velocity estimate with covariance matrix for each cell resulting in a spatial occupancy and velocity distribution. For this, the algorithm uses advanced sensor fusion techniques regarding latency and variance in time domain which are hard to compensate by learning based approaches [8].
2. The DOGMa output format is independent of the sensor setup, e.g., fusing radar and laser or only lasers with different range, which can be advantageous when training a neural network with varying sensors. In general, the algorithm exploits sensor dependent strengths using engineered models. These models can, e.g., consider free space gained from a laser measurement, or ego motion compensated Doppler velocities from radars [16]. If sensor specifications change, parameters can easily be adapted.

The DOGMa is based on Sequential Monte Carlo filtering. Grid cells have a static position and width, and the map border is shifted by the cell width to keep the moving ego vehicle within the center cell. The filter algorithm runs in two modules: In the first, sensor data is converted into classical occupancy grid maps [17, 18] using sensor specific measurement models. These grids serve as a generic data interface to the second module, a particle filter estimating the spatial occupancy and velocity distribution [16]. Each grid cell is updated independently and no interaction between cells is modeled. Although, hand-crafted methods were proposed to extract modeled objects from grid cells [19], in this work, no object generation is needed at the CNN input or for labeling. Due to the implementation in the Dempster-Shafer domain [20], a DOGMa provides probabilistic information about occupancy, free space, and unobservable areas.

III Automatic Output Label Generation

Labeling is usually an extensive process and one of the major drawbacks in supervised learning. In the nature of scene prediction, fortunately, the desired algorithm output can be observed at a later time. Once automatic label extraction can be applied, it is comparably easy to generate training data. However, about $99.75\%$ of our environment data is static, which might lead the learning algorithm to favor the trivial result of predicting the future to be equal to the current input sample. For this reason, a segmentation of the underrepresented dynamic environment is required to apply balancing methods. Methods to classify dynamic cells online at perception time are proposed by [16]. However, using the current cell velocity estimate leads to misclassification when static objects enter the field of view or at noisy regions, e.g., the borders of static objects. Since the label extraction can be done offline, a better classification can be achieved using future and past occupancy course, and thus, e.g., distinguish between growing static parts and actually moving objects.

For each cell, a quasi time continuous $P_{\mathrm{O}}(t)$ is extracted, as illustrated in Fig. 2. The figure illustrates the extraction of $P_{\mathrm{O}}(t)$ for grid cells at constant spatial coordinates $(E,N)$, while the ego vehicle is itself moving in a dynamic environment. The desired prediction result is a sequence

[TABLE]

where the time step sequence $k=(1,2,...)$ can be represented in corresponding output channels of the CNN. In addition, a segmentation into static and dynamic parts is desired. Thus, one can write

[TABLE]

with a constant occupancy probability $P_{\mathrm{O},\mathrm{s}}$ of the static environment and a sequence $\left(P_{\mathrm{O},\mathrm{d}}(k)\right)$ describing the occupancy probability originated solely from dynamic elements. The segmentation can be done automatically in time domain by dividing data cell-wise in static and dynamic time $t_{\mathrm{s}}$ and $t_{\mathrm{d}}$: When objects traverse a cell, the Bayesian filter in the DOGMa converges smoothly to a maximum $P_{\mathrm{O}}$ and back to the static level when the object leaves the cell (see Fig. 2). A rise in $P_{\mathrm{O}}(t)$ followed by a drop of the signal is used to detect the presence of a dynamic object, and thus the time in between defines the interval $t_{\mathrm{d}}$. This calculation is based on the second derivative followed by non-maximum suppression. In the offline extracted sequence, the DOGMa filter convergence delay is corrected by setting $\left(P_{\mathrm{O}}(k)\right)$ to $\max\left(P_{\mathrm{O}}(t_{\mathrm{d}})\right)$ for time steps within $t_{\mathrm{d}}$. The curve $P_{\mathrm{O}}(t)$ can be quite noisy, especially in unknown regions ($P_{\mathrm{O}}=0.5$) or at the spatial transition between static, free, and unknown. Therefore, the data in $\Omega$ is smoothed with a Gaussian filter in time domain. Noise in spatial domain, e.g., a single dynamic cell, is smoothed accordingly.

Finally, the constant static $P_{\mathrm{O},\mathrm{s}}$ in (3) is calculated by the median of $P_{\mathrm{O},\mathrm{s}}(t_{s})$ with $t_{s}\in(t_{0},t_{0}+T)\setminus t_{d}$, where $t_{0}$ denotes the DOGMa input time and $T$ the prediction horizon. The network output in $\mathbb{R}^{|\mathcal{T}|\times W\times H}$ provides the channels $\mathcal{T}=\left\{P_{\mathrm{O},\mathrm{s}},P_{\mathrm{O},\mathrm{d}}(1),P_{\mathrm{O},\mathrm{d}}(2),...,P_{\mathrm{O},\mathrm{d}}(|\mathcal{T}|-1)\right\}$.

IV CNN Architecture

A deep neural network architecture can exploit far distant relations within the DOGMa input. However, the output structure should also provide a spatial distribution. The network architectures of [21] (DeconvNet) and [22] (Fully Convolutional Networks, FCN), provide upscaling layers following downscaling cascades. These approaches aim at pixel precise image segmentation from a single input image. The group around Shelhamer and Long [22, 23] resizes down sampled feature maps using deconvolution operations proposed by Zeiler and Fergus [24] for visualization purposes. Upsampling is performed in a single layer while earlier layer results with a higher resolution are used to gain a fine output map. In contrast to learnable kernels, the authors of [21] and [25] use the unpooling method of [26]. While [22] use a single upsampling step, [21] perform upscaling stepwise, mirroring the downsampling stages and reusing the indices of max-pooling downscaling for upscaling by unpooling. The resulting sparse feature maps are made dense by subsequent convolution in each unpooling layer. In our approach, we follow the strategy of a single input DOGMa processed by a downscaling cascade mirrored by upscaling stages, as proposed by [21]. However, we chose learnable deconvolution kernels instead of unpooling, combined with bypassed results from according downscaling stages.

Instead of pixel classification we aim at the prediction of occupied cells at future time steps. Moreover, the network segments static and dynamic parts of the environment and delivers the occupancy of static grid cells in one channel and the occupancy of dynamic cells in the remaining channels.

V Spatial Balancing Loss

Due to the segmentation of the labels in static and dynamic areas, balancing methods can be applied to weight rare ($0.25\%$) dynamic cells more compared to static cells. In our specific case, the network is also trained to perform a segmentation of static and dynamic regions. Therefore, the occupancy information of static cells is stored in the first channel, while only occupancy of rare dynamic cells is included in the remaining time channels. The time channel labels of static cells are set to [math]. For one sample, the loss

[TABLE]

is the sum of a static channel loss $L_{\mathrm{s}}$ and the loss of dynamic channels $L_{\mathrm{d}}$. The linear least squares of the single static output channel in $\mathcal{T}$ is used to calculate the loss of static occupancy estimation for each grid cell $c$ with

[TABLE]

with $\lambda_{\mathrm{s}}$ denoting a static weighting factor. We write the labeled pixel value with $P_{\mathrm{O},.}$ and the network output $\hat{P}_{\mathrm{O},.}$.

For the dynamic channels we calculate a cell wise weighting factor

[TABLE]

for each time channel $k$ and grid cell $c$. Since $P_{\mathrm{O},\mathrm{d}}(c,k)\in\left[0,1\right]$ with $P_{\mathrm{O},\mathrm{d}}(c,k)=0$ at static cells, $\lambda_{c,k}=1$ for static and $\lambda_{c,k}=1+\lambda_{k}$ for dynamic regions. The factor $\lambda_{k}$ is used to apply increasing weighting factors for the prediction time channels $k$. The dynamic loss is then calculated with

[TABLE]

and contains an inherited discrimination between static and dynamic. For back propagation, the partial derivative of (7) at a cell $c$ and time channel $k$ is calculated with

[TABLE]

VI Dataset and Training

We recorded about $2.5\text{\,}\mathrm{h}$ of an urban shared space scene with pedestrians, bikes and motor vehicles at three different days. At every day, the recording vehicle was placed at two different positions, facing west and east, as illustrated in Fig. 3. The update rate of the Dynamic Grid Map is about $100\text{\,}\mathrm{m}\mathrm{s}$ resulting in $77582$ samples in total. We used $62800$ samples for training, $7590$ for testing and $7192$ for validation.

The ADAM solver [27] was chosen for training, due to its automatic learning rate adjustment. As suggested in [27], we chose the exponential decay rates to $\beta_{1}=0.9$ and $\beta_{2}=0.999$. The base learning rate was chosen to $l=0.0001$. The training process was stopped after about $1.6$ epochs ($100000$ iterations).

Input space reduction to gain computation speed and save memory is very common in machine learning approaches with image data. However, simply down scaling the covariance matrices in DOGMa cells is formally wrong, due to the assumption of independent cell objects [16] in the fusion approach. Furthermore, velocity differences in neighboring cells represent a local velocity variance, which can be exploited by the convolutional network architecture. This makes the single cell covariance redundant, which is stored in the last three channels in $\Omega$. Therefore, we chose to reduce the input space by omitting the velocity covariance matrices and use only the channels $M_{\mathrm{O}}$, $M_{\mathrm{F}}$, $v_{\mathrm{E}}$, and $v_{\mathrm{N}}$. The prediction target times were chosen to 0.5\text{\,}\mathrm{s}$,$1\text{\,}\mathrm{s}$,$1.5\text{\,}\mathrm{s}$,...,$3\text{\,}\mathrm{s}.

VII Evaluation

The network was trained and tested in a downtown scenario with numerous dynamic objects. Due to the shared space situation where pedestrians and bikes cross the road at arbitrary locations, the prediction task is particularly challenging. The location also confronts us with occlusions in the field of view. The experimental vehicle provides $360\text{\,}\mathrm{\SIUnitSymbolDegree}$ perception using an IBEO LUX long-range laser scanner covering about $200\text{\,}\mathrm{m}$ in the front with an opening angle of about $100\text{\,}\mathrm{\SIUnitSymbolDegree}$ and four Velodyne VLP-16 laser scanners, each providing $16$ layers laser measurement ranging up to $100\text{\,}\mathrm{m}$. The DOGMa covers $901\times 901$ cells with a cell width of $0.15\text{\,}\mathrm{m}$. To evaluate the prediction, we used about $12\text{\,}\mathrm{m}\mathrm{i}\mathrm{n}$ from $4$ sequences. Videos showing the prediction results can be watched under https://youtu.be/iw-eXnnjfj0.

We compare the learning based prediction to a particle based algorithm. The occupancy probability and velocity distribution in $\Omega$ of a grid cell $c$ is represented by particles with state vector $\mathbf{x}=\left[E,N,v_{\mathrm{E}},v_{\mathrm{N}}\right]^{\mathrm{T}}$. We calculate a long-term prediction by forward propagating the particles with the constant velocity model. Only particles of non-static cells are forward propagated. Cells are considered non-static if the velocity variance is below $3\text{\,}\mathrm{m}^{2}\mathrm{/}\mathrm{s}^{2}$ and the velocity magnitude is greater than $0.7\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$. These parameters turned out to yield good separation between static and dynamic regions, however, their heuristic nature is a drawback compared to the learning based approach. In our experiments, $900000$ particles were sampled and forward propagated at dynamic cells to provide a dense distribution of predicted occupancy.

The overall prediction performance is illustrated in Fig. 4 showing a receiver operating characteristic (ROC) curve for the learned and particle approach with different prediction times. For this plot, each dynamic cell is classified as occupied or free with the varying threshold parameter $\gamma\in(0,1)$. It is important to note, that the two approaches are not compared against automatically generated labels, but to the actually perceived DOGMa at the corresponding prediction time including static and dynamic cells. To calculate true positive and false negative rates in Fig. 4, the perceived DOGMa is thresholded with $P_{\mathrm{O}}>0.55$, using a fixed value slightly over the ambiguous value of $0.5$. Also, static parts in the prediction approaches were thresholded with this fixed value, while the varying threshold is only applied to dynamic prediction. The dynamic neural network channels are directly thresholded with $\gamma$, while for the particle approach the prediction is considered occupied if the number of particles per cell $n_{c}$ is greater than $\gamma\cdot\max(n_{c})$.

In general, it can be observed, that for an increasing prediction time, a growing false positive rate (fpr) has to be taken into account to reach a common true positive rate (tpr), which means that the uncertainty of the predicted area grows with increasing prediction time. The curve start points at $\mathrm{fpr}=0$ illustrate the case when dynamic prediction is completely ignored. At this point it can be seen, that in the particle approach, a minimum tpr of $0.75$ can already be reached by just predicting static cells. The learning based approach, however, reaches a minimum tpr of $0.81$, indicating better performance for classifying static regions in the grid map.

The top right ends of the curves illustrate the ROC for $\gamma=0$, while the cases $n_{c}=0$ (no particle in a cell) and $P_{\mathrm{O}}=0$ are not considered occupied due to the grater condition. The limitation of the tpr for both approaches means, that false negatives cannot be eliminated. Examples are given below. However, the maximum tpr at a prediction time of $0.5\text{\,}\mathrm{s}$ lies at $98\%$ for the learning based approach and $89\%$ for the particle approach. A longer prediction time causes higher uncertainty which leads to a growing area covered by particles or predicted occupied by the network. This in turn leads to an increasing true positive rate at the expense of a growing false positive rate. Thus, the maximum true positive rate for the particle filter lies at $95\%$ at $3\text{\,}\mathrm{s}$ prediction horizon and all particle based ROC curves never reach the worst ROC curve of the learning based approach.

VII-A Predicting Multi-Modal Scene Evolution

Fig. 5 shows an example of long-term prediction results in a complex scenario. More specifically, it depicts the image of the on-board camera, showing parallel traffic with a possible right-hand turn for oncoming traffic. The bottom left image in Fig. 5 shows the vehicle surrounding represented by a DOGMa. The cell colors and overlaid orange lines visualize the filtered velocity of moving cells. The displayed DOGMa is used as only input to the prediction network. The following RGB images in the bottom row visualize the prediction result at times $0.5\text{\,}\mathrm{s}$ to $3.0\text{\,}\mathrm{s}$. The color channels in these images encode the different approaches. The true occupancy is encoded in the red channel, the output of the neural net appears in the green channel, and the particle prediction is illustrated in the blue channel. An overlap of the prediction leads to a mixed RGB-color, ideally white when both approaches overlap with true occupancy, as for example at most static regions. For better visualization, static cells are only drawn if $P_{\mathrm{O}}>0.6$.

Note, that at the given time instance, it is not obvious whether the vehicle marked with a red ellipse will be taking a right-hand turn or pass the junction. Hence, the prediction network outputs a bimodal occupancy distribution (green) for the vehicle’s trajectory, while propagated particles (blue) only cover the straight maneuver, based on the initial velocity distribution. This can be seen in Fig. 5 within the prediction horizon $1.0\text{\,}\mathrm{s}$ to $3.0\text{\,}\mathrm{s}$. The RGB color appears yellow if only the learned prediction overlaps with the true occupancy. These results indicate that the prediction network is capable of inferring possible driving maneuvers and even the spatio-temporal occupancy distribution of future trajectories.

VII-B Static Regions

Distinguishing between static and dynamic cells is a hard task considering only cell-wise features. The particle prediction, as well as the automatic label generation algorithm contain misclassifications of static cells, especially at the border of static regions. In the particle based prediction, misclassification due to bad parameters leads either to blurred occupancy prediction of static areas, or to missing predictions of dynamic objects. An example can be seen in Fig. 5. The corners of the static wall are surrounded by predicted particles, drawn in blue. The neural network in contrast, can deal with misclassified cells in the automatic labeling process by simply not predicting the movement of static regions. However, the static waiting pedestrian in Fig. 6 is predicted to possibly cross the street by the learning based approach, while the particle approach can only predict its movement as soon as it moves.

VII-C Predicting Partially Occluded Objects

Another pattern, typical for dynamic occupancy grids, is that for objects entering the field of view, the area estimated as occupied tends to grow. From initially only a few occupied cells, a solid silhouette can be seen when the object is deeper in the field of view and the filter converges. The same applies for vanishing occlusion. The neural network learned to predict an enlarged occupied area, even when only a few pixels are initially occupied and dynamic in the DOGMa. The last row in Fig. 6 shows a vehicle driving upwards in the DOGMa input image which is partially occluded by a downwards driving vehicle. The prediction result of the neural network (green), covers the entire vehicle, while the particle approach (blue) only predicts the vehicle front, underestimating the future occupied area. Object detection could improve the particle approach, considering object size and occlusions. However, object detection can be challenging when objects just enter the field of view [19].

VII-D Reducing Prediction Variance with Object Interactions

A main motivation for the use of a CNN is the ability to exploit spatial context, an advantage to model object interactions. Interactions reduce the future trajectory space, e.g., due to social forces [28] or traffic rules. An example can be observed in the car following scenario illustrated in Fig. 6. The figure includes two moving vehicles: the left one in the input DOGMa, appearing purple, drives downwards approaching a standing car. The right one, appearing light green in the input DOGMa and marked with a red dashed ellipse, drives upwards on a free lane. Prediction on the free lane is very uncertain after $2\text{\,}\mathrm{s}$, resulting in a large area predicted to possibly be occupied. On the blocked lane, in contrast, the network predicts the vehicle to slow down, reducing the future scene variance and causing a smaller region to be predicted as occupied. The particle filter overestimates the vehicle’s motion, resulting in a collision with the standing object.

The scene also illustrates an example of pedestrian-vehicle interactions. A pedestrian stands close to the road, his intention is not clear, and the vehicle marked with the red ellipse is approaching from his right-hand side. The last two rows illustrate the scene prediction at two different starting times. In the first prediction, the approaching vehicle is still about $3\text{\,}\mathrm{s}$ away from the pedestrian, and a possible crossing trajectory is predicted by the neural network. Starting the prediction $1.8\text{\,}\mathrm{s}$ later (bottom row), the pedestrian still didn’t move, but the approaching vehicle came closer. For the pedestrian it would be hard to cross the road now, since the vehicle arrives at the pedestrian in about $1\text{\,}\mathrm{s}$, true and predicted. Appropriately, the neural network prediction for the pedestrian adapted and no crossing trajectory is predicted anymore. The particle approach, in contrast, cannot predict any movement for temporarily static objects.

The ability of the trained CNN to model interactions is further investigated in the following experiment. Rather than using only real-world data with a single observable outcome, manipulating the input can yield useful information on the predictive power of the CNN. Therefore, we added objects to a given scene in order to force the model to react on the new scenario. Fig. 7 shows side by side the prediction of two almost identical scenes of an upwards driving vehicle. In the right column, however, a pedestrian was artificially added off-road approaching the lane. The blue dash-dotted ellipse highlights the input modification. The insertion was accomplished by extracting the DOGMa representation of a pedestrian from another recording and placing it into the scene of interest. Horizontal reference lines (red) illustrate the difference in occupancy prediction at points of interest. The predictions of the upwards driving car in the original scenario do not show any explicit interactions with objects in its environment, and is consequently used as reference occupancy to study the role of interactions. By examining the predictions at $1.0\text{\,}\mathrm{s}$, in case of the augmented input, the car already does not advance as much as in respect to the original case. This predicted deceleration of the vehicle, can only be a result of interaction with the added pedestrian. Inspecting the further prediction results, the effect becomes even more obvious. At $1.5\text{\,}\mathrm{s}$, the original prediction of the vehicle already exceeds the spatial position of the pedestrian included in the other case, which would lead to a collision if interaction is not modeled. But on the contrary, the network reacts to the presence of the pedestrian by a slowed down predicted movement, which can be perceived by the less likely occupancy in the area between the car and the pedestrian ($1.5\text{\,}\mathrm{s}$). Most obviously, the vehicle will not run over the pedestrian, causing no occupancy behind it, while, in the case of non-existence, the car is predicted to just go through ($2.0\text{\,}\mathrm{s}$). In conclusion, in the augmented scenario, the predicted occupancy caused by the vehicle alters, which can only be due to the particular presence of the velocities and occupancies of the pedestrian.

VIII Conclusions

We presented a learning based situation prediction approach predicting the $360\text{\,}\mathrm{\SIUnitSymbolDegree}$ perceivable scene in a single neural network. Beside prediction, the network performs a segmentation of static and dynamic areas. The machine learning approach benefits from a fully automatic label generation method, yielding an unsupervised character. We introduced a loss function that counteracts imbalanced pixels of different categories and trained the dataset with rare dynamic cells. The prediction was compared to a particle based approach. The results showed, that the network can predict highly complex scenarios with various road users of different class up to $3\text{\,}\mathrm{s}$. It can be seen, that the network is able to consider different maneuver classes, e.g., turn right or go straight, and interactions between road users reduce the prediction uncertainty. In ongoing studies, the loss function is extended to consider the probabilistic nature of prediction by adjusting the pixel-wise weighting factors according to increasing variance of an object’s future position. Experiments with other network architectures, i.e., recurrent neural networks are planned.

Acknowledgements

The research leading to these results has received funding from the European Union under the H2020 EU.2.1.1.7. ECSEL Programme, as part of the RobustSENSE project, contract number 661933. Responsibility for the information and views in this publication lies entirely with the authors. The authors would like to thank all RobustSENSE partners.