Deep, spatially coherent Inverse Sensor Models with Uncertainty Incorporation using the evidential Framework

TL;DR

This paper introduces an evidential neural network framework for radar-based environment perception in autonomous cars, enabling dense, spatially coherent, and uncertainty-aware inference from sparse, noisy radar data, thus improving perception speed.

Contribution

It extends evidential convolutional neural networks to incorporate sensor and model uncertainty for inverse sensor models, enhancing environment perception in autonomous vehicles.

Findings

Denser environment perception achieved in fewer time steps.

Effective incorporation of sensor noise and model uncertainty.

Improved spatial coherence in environment inference.

Abstract

To perform high speed tasks, sensors of autonomous cars have to provide as much information in as few time steps as possible. However, radars, one of the sensor modalities autonomous cars heavily rely on, often only provide sparse, noisy detections. These have to be accumulated over time to reach a high enough confidence about the static parts of the environment. For radars, the state is typically estimated by accumulating inverse detection models (IDMs). We employ the recently proposed evidential convolutional neural networks which, in contrast to IDMs, compute dense, spatially coherent inference of the environment state. Moreover, these networks are able to incorporate sensor noise in a principled way which we further extend to also incorporate model uncertainty. We present experimental results that show This makes it possible to obtain a denser environment perception in fewer time steps.

Figures

Tables

Table 1. TABLE I: scores for visible area

$[\%]$	$p_{\left(\hat{f}|\tilde{f}\right)}$	$p_{\left(\hat{o}|\tilde{f}\right)}$	$p_{\left(\hat{u}|\tilde{f}\right)}$	$p_{\left(\hat{c}|\tilde{f}\right)}$	$p_{\left(\hat{f}|\tilde{o}\right)}$	$p_{\left(\hat{o}|\tilde{o}\right)}$	$p_{\left(\hat{u}|\tilde{o}\right)}$	$p_{\left(\hat{c}|\tilde{o}\right)}$
Ray-ISM	25	0	75	0	25	4	71	0
Soft-Net	76	16	8	19	17	71	12	19
Ev-Net	64	3	33	1	8	20	72	0
Ev-Net-S	48	0	52	0	3	2	95	0

Equations

$${\mathbf{s}}^{(p)}$$

$${\mathbf{s}}^{(e)}$$

$$\mathbf{b}$$

$$\alpha$$

$$\mathbf{p}$$

$$\mathcal{L}(\theta {)}_i$$

$$\approx \sum _{k=f,o}({\tilde{p}}_{i,k}-{\hat{p}}_{i,k}{)}^2\cdot \frac{{\hat{p}}_{i,k}(1-{\hat{p}}_{i,k})}{S_i+1}$$

$$\displaystyle\mathcal{L}(\theta)_{i}=\Bigg{(}\sum_{k={f,o}}$$

$$+(1-{\hat{u}}_i{)}^2\cdot {\tilde{u}}_i$$

$${\left[\begin{array}{cc}{p}_f=0,& p_o=1\end{array}\right]}^{\top }(r,\phi )=:{\mathbf{s}}_o^{(r,\phi )}$$

$$p(s_o^{(r,\phi )}|\tilde{r},\tilde{\phi })$$

$$={\iint }_{R,\Phi }p(s_o^{(r,\phi )}|R,\Phi )p(R|\tilde{r})p(\Phi |\tilde{\phi })dRd\Phi$$

$$p(s_o^{(r,\phi )}|\tilde{r},\tilde{\phi })$$

$$p(s_o^{(r,\phi )}|$$

$$+{\int }_{\Phi }(p(s_o^{(r)}|\tilde{r})-0.5)\delta (\phi -\Phi )\mathcal{N}(\Phi |\tilde{\phi },{\sigma }_{\phi })d\Phi$$

$${\int }_x\mathcal{N}(x|\mu ,\sigma )dx\equiv 1\text{ and }{\int }_x\delta (x-a)f(x)dx\equiv f(a)$$

$$p(s_o^{(r,\phi )}|\tilde{r},\tilde{\phi })$$

$$p(\hat{u}=\max ({\hat{b}}_f,{\hat{b}}_o,\hat{u})|{\tilde{b}}_j)=:p_{(\hat{u}|j)}\text{, }j=\{\tilde{f},\tilde{o}\}$$

$$p({\hat{b}}_i=\max ({\hat{b}}_f,{\hat{b}}_o,\hat{u})|{\tilde{b}}_j)=:p_{(i|j)}\text{, }i=\{\hat{f},\hat{o}\}$$

$$p(|{\hat{b}}_f-{\hat{b}}_o|\le 0.2|{\tilde{b}}_j)=:p_{(\hat{c}|j)}$$

Full text

I INTRODUCTION

The estimation of traversable and occupied areas in an environment is a necessity to reach the goal for cars to behave autonomously. In order to do so, modern cars rely on the combination of different sensors that complement each other. Because of its robustness to environmental changes and its low production price, radars are a major part of current automotive sensor suits [1]. To obtain the information of traversable and occupied areas based on the measurements, an inverse sensor model (ISM) has to be applied. The typical way to obtain the ISM would then be to first apply an inverse detection model (IDM) which describes the state of the environment solely based on a single detection and afterwards, accumulate the IDM estimates [2]. Since the summation operation is commutative, the detection’s order and thus their spatial correlation gets lost in this process. Additionally, capturing all environment dependent effects for sensors like radars is a non-trivial task. A typical radar IDM and the resulting ISM for the free, occupied and unknown state is shown in the upper part of Fig. 1.

Because of the uncertainties and the sparseness in the radar data, many measurements have to be accumulated over time to reach certain levels of confidence about the environment. Thus, using sparse radar detections for environment modeling at high velocities for e.g. on highways becomes a difficult task.

In this work, we address the sparsity problem and the modeling of environment dependent sensor effects of the above explained radar ISM by employing evidential-based convolutional neural networks [3]. In order for the model to learn these relationships, we use patches cut out of LiDAR-based occupancy maps according to the vehicles position as targets during training. By doing so, we try to approximate the true state of the environment around the vehicle as close as possible. Moreover, evidential convolutional neural networks are able to decrease the classification probabilities according to the uncertainties in a dataset and shift the reduced amount into a separate unknown state in an end-to-end manner. We further extend this framework to incorporate model uncertainties and show how this improves the robustness of the predictions in contrast to convolutional neural networks without the evidential extension.

II RELATED WORK

In this work, we want to describe the occupancy state in a squared area around a vehicle which we will refer to as a scene. A common representation of the scenes states for real-time robotics ([4, 5, 6]) is a discrete 2D bird’s-eye-view grid. To describe the occupancy, there are currently two main frameworks, namely the probabilistic framework [7] and the Dempster-Shafer theory of evidence (DST) [8, 9]. Here, the state can either be expressed by the probability of being free/occupied $p_{f}\in[0,1]$/$p_{o}=1-p_{f}$ or alternatively by the belief of being free, occupied or unknown $b_{f},b_{o},u\in[0,1]$, $b_{f}+b_{o}+u=1$. By separately modeling the unknown state, the DST is able to express a conflict state additionally to the unknown state. This additional information can aid in tasks like moving object detection, which motivates us to use the DST [10].

II-A Inverse Sensor Models based on Inverse Detection Models

To update a discrete occupancy map given radar measurements, an IDM is typically used. Such an IDM is depicted in Fig. 3 (left) for a noise-free detection of a radial sensor like camera, radar or LiDAR. In the event of uncertainty, several approaches have been proposed. Werber et al. [11] modeled the occupancy state around a detection as Gaussian distribution while all the points in a straight line between the uncertainty ellipse and the sensor are free to a certain extent. This approach, however, results in discontinuous behavior at the boundary between occupied and free space. An alternative was proposed by Loop et al. [12]. It describes the incorporation of sensor noise characteristics in a principled way using the Bayesian framework. This is why we propose to use it as a reference for IDMs in this work. A more detailed description is provided in sec. V-A.

II-B Deep Inverse Sensor Models

Alternatively to using an IDM, the occupation state can be estimated using all detections and their correlations at the same time. Because of the high complexity of such models, they are dominated by deep learning approaches. Training neural networks to learn sensor characteristics has a long history [13, 14]. However, with the recent progress in stochastic inference in deep learning, models have been proposed to also incorporate uncertainties. Wheeler et al. [15] addressed training of forward radar models which estimate the radar signal given the state of the environment. To do so, they used a conditional variational Autoencoder (VAE) [16] additionally trained with an adversarial loss [17] to reduce blur. Lundell et al. [18] proposed a model to estimate the distance to each untraversable region insight as a Laplace probability density function (PDF). Moreover, they described how to integrate the Laplace distance PDF into an IDM. Recently, Weston et al. [19] used a VAE like architecture to infer the state of the environment based on raw, dense radar images. In contrast to the traditional VAE, they modeled the log odds to be Gaussian distributed. To incorporate the uncertainties, they treated the Gaussian log odds as a prior to a Sigmoid and marginalized out the uncertainty.

In this work, we approach the problem from an evidential view and hence need to reformulate the uncertainty estimation and incorporation. However, the majority of proposed deep ISMs use a U-Net architecture with skip connections as a model [20], which we also adapt for our work.

II-C Uncertainty estimation in Neural Networks

The uncertainty estimation methods mentioned in Sec. II-B model two types of uncertainties, namely the model and data uncertainties which are often referred to as epistemic and aleatoric [21]. Epistemic uncertainties can be estimated in neural networks using sampling techniques e.g. perturbation of weights, inputs or activations [22, 17, 23] or by using ensemble methods [24, 25]. These estimates can e.g. be used to predict how closely a new data point resembles the training set and to assess the robustness of predictions during test time. Aleatoric uncertainties, on the other hand, directly estimate the shape parameters of an a priori defined probability distribution e.g. the mean and variance of a Gaussian [26, 18]. One method particularly similar to ours has been proposed by Sensoy et al. [3]. They transform an aleatoric, probabilistic classification model into an evidential one using Subjective Logic [27].

We propose to adapt Sensoy’s evidential approach for datasets with explicit unknown regions and extend it to use both aleatoric and epistemic uncertainties.

III METHOD

In this work, we seek to infer the occupancy state of a scene $\mathbf{s}$ given radar measurements $\mathbf{x}$. This state can be expressed in a probabilistic (1) or evidential (2) frame as follows

[TABLE]

with $p_{f},p_{o}$ being the free and occupation probabilities respectively, $u$ the unknown mass and $b_{f}$, $b_{o}$ the free and occupation belief masses respectively. We want to compare the IDM-based ISM with two deep learning approaches.

The first deep learning method consists in treating the problem as a standard three class classification task. Here, we can train a neural network $f_{\phi}$ to estimate a belief vector which is as close as possible to a target vector $\tilde{\mathbf{s}}$ given radar inputs. This, however, treats the unknown state as a separate class which neither models nor incorporates the uncertainties of the free and occupied predictions. To ensure comparability, we also apply an epistemic approach to $f_{\phi}$ and use the expectation $\hat{\mathbf{y}}$ as an input to the softmax. This architecture is depicted in Fig. 2 (top).

The second method we want to investigate follows Sensoy et al. [3]. The method is described for the general case of $K$ classes with an additional unknown class leading to a $K+1$ class classification problem. At the beginning, the unknown state is ignored and a Dirichlet density network [28, 3] is used to model the PDF over the remaining $K$ classes. Hence, we model the second order probabilities of our classes which can be used to state how certain the classification probabilities are. More specifically, the Dirichlet distribution’s expectation and variance can be interpreted as the classification result and its corresponding aleatoric uncertainty estimate. This is illustrated in the orange boxes on the right in Fig. 2 for two examples which have the same classification probabilities with different uncertainties.

The remaining question is how to incorporate the additional information about the uncertainty into the classification results. Here, we seek to utilize the uncertainties by reducing each classification probability accordingly and collect all the uncertainty mass in the so far ignored unknown class. This corresponds to a transformation from probabilistic to evidential view which can be described by Subjective Logic [27] in a principled way.

III-A Subjective Logic

In Subjective Logic, evidence $\mathbf{e}\in\mathbb{R}_{>0}$ is collected for each of the $K$ classes in the form of strictly positive, real numbers. Given such an evidence vector, Subjective Logic defines the corresponding class beliefs $\mathbf{b}$ and the unknown state $u$ as follows

[TABLE]

On the probabilistic side, the classification probabilities correspond to the expectation of a Dirichlet PDF whose shape parameters $\mathbf{\alpha}$ depend on the evidence in the following way

[TABLE]

Here, the base rate $\mathbf{a}\in[0,1]$, $\sum_{k=1}^{K}a_{k}=1$ is the connecting element between the evidential and probabilistic view. It defines how the unknown mass shall be distributed into the $K$ probabilities in the following way

[TABLE]

We chose $\mathbf{a}=\dfrac{1}{K}\text{I}$ since we want the probabilities to be equally distributed in the case of complete unknowingness.

We can now use a neural network $f_{\theta}$ to estimate an evidence vector by e.g. using a quadratic output activation function $\hat{\mathbf{e}}=f_{\theta}(\mathbf{x})^{2}$. This vector can then be used to define the shape parameters of a Dirichlet distribution (4). The network can then be trained in a way that the Dirichlet’s expectation resembles the training data as closely as possible. During test time, the neural network’s evidence estimates can be used to compute the belief masses following (3).

III-B Epistemic Uncertainties

Now we want to describe a method of how to incorporate epistemic uncertainties into this framework. We address this problem by modeling the network’s evidence estimates as an epistemic PDF. To do so, any of the methods mentioned in sec. II-C can be applied. The way we want to make the estimations more robust using the epistemic PDF is by taking the $n$th percentile instead of the expectation as the prediction. This converges to the secure unknown state in case of high uncertainty. Moreover, we do not make any assumptions on the PDF’s shape. Since epistemic uncertainty prediction methods currently highly rely on sampling, the $n$th percentile can simply be found by sorting the samples (e.g. for $100$ samples the $10$th percentile corresponds to the $10$th smallest sample). In the following, we will refer to the epistemic PDF’s expectation as $\hat{\mathbf{y}}$ and to the $n$th percentile estimate as $\hat{\mathbf{y}}_{s}$. A complete illustration of the model is depicted in Fig. 2.

III-C Training the Model

Finally, we want to address how to train a neural network with a Dirichlet output activation. Several approaches have been proposed e.g. to minimize the Dirichlet’s negative log-likelihood [28] or the Bayesian risk [3]. We will use the Bayesian risk with a mean squared loss and Dirichlet prior as shown in (7) since it has been reported to perform with greater stability [28].

[TABLE]

Here $\mathcal{L}(\theta)_{i}$ is the loss of the $i$th pixel of one training sample, $\tilde{\mathbf{p}}_{i}$ are the target class probabilities and $\hat{\mathbf{p}}_{i}$ is the expectation of the Dirichlet. This loss ensures that $\hat{\mathbf{p}}_{i}$ is as close as possible to $\tilde{\mathbf{p}}_{i}$ while at the same time maximizing the accumulated evidence $S_{i}$ which reduces the uncertainty. Sensoy et al. [3] have shown that for this loss the reconstruction part is always favored over the decrease of uncertainty.

III-D Unknown Observations

The problem with the above formulation is that an unknown observation cannot be expressed correctly as a probability. The closest possibility would be to describe the unknown state as $\tilde{\mathbf{p}}=\begin{bmatrix}0.5&0.5\end{bmatrix}^{\top}$. However, the network is able to predict areas where the unknown state is caused by occlusion with high certainty. This leads to an increase in evidence being uniformly distributed into both the free and occupied class. Thus, rather than predicting the unknown targets as $\begin{bmatrix}\hat{\mathbf{b}},&\hat{u}\end{bmatrix}^{\top}=\begin{bmatrix}0,&0,&1\end{bmatrix}^{\top}$ the network ends up predicting them to be conflicting areas $\begin{bmatrix}\hat{\mathbf{b}},&\hat{u}\end{bmatrix}^{\top}=\begin{bmatrix}0.5,&0.5,&0\end{bmatrix}^{\top}$. We therefore altered the loss in the following way

[TABLE]

The first change is that we used the belief masses as targets. Moreover, we introduced a second term that reduces the overall estimated evidence in case of an unknown target by maximizing the unknown state’s mass. As targets we allow free $\begin{bmatrix}1,&0,&0\end{bmatrix}^{\top}$, occupied $\begin{bmatrix}0,&1,&0\end{bmatrix}^{\top}$, conflict $\begin{bmatrix}0.5,&0.5,&0\end{bmatrix}^{\top}$ or unknown $\begin{bmatrix}0,&0,&1\end{bmatrix}^{\top}$. For the first three targets, the loss collapses back to its original version. However, for the fourth case, the first term gets canceled out and the second term is active. This enables the network to utilize unknown observations during training.

IV EXPERIMENTAL SETUP

IV-A Data Collection

To collect the data, a roof-mounted Velodyne HDL-32 LiDAR, four short range, automotive radars each mounted at a corner of the car and the vehicle’s dead-reckoning system are used as sensor inputs. The radars are capable of detecting range, angle and relative velocity of up to 64 objects. The driven route resulted in $28843$ training, $4775$ test and $4775$ validation images. During training, the images are additionally augmented by randomly flipping them along the horizontal and vertical axis and rotating them by a random multiple of $90^{\circ}$.

IV-B Radar Images

The radar images are used as inputs for the neural networks. To create them, we used all the detections of the four corner radars falling into a square area around the vehicle. These detections were transformed and discretized into Cartesian pixel coordinates. Moreover, to account for moving objects, the detections with an absolute velocity higher than a certain threshold were identified as dynamic objects and moved into a second layer. Hence, the radar images are of the dimension $128\times 128\times 2$ with the first layer containing the static and the second layer the dynamic detections.

IV-C LiDAR-based Ground Truth Occupancy Patches

To obtain targets for the training, LiDAR-based occupancy maps were created. In this process, the LiDAR detections were transformed and discretized into Cartesian pixel coordinates and an ideal radial ISM was applied as shown in Fig. 3 (right). As a simplification, only the non-occluded detections were used. The occupancy estimates of each scene were aligned and accumulated using the pose estimates of the vehicle’s dead-reckoning system. Afterwards, for each radar image, a patch was cut out of the LiDAR occupancy maps covering the same area. These patches are then used as approximations of the scene’s true occupancy beliefs by discretizing the estimates into free, occupied and unknown pixels resulting in images of dimension $128\times 128\times 3$. Using the occupancy map patches as opposed to the LiDAR ISM enables the neural network to learn to predict the occupancy in occluded areas based on shape primitives in the data, for example the typical L-shape of partially observed cars.

V MODEL

In this section, we will describe the different approaches to create the ISMs that we will compare in this work.

V-A Reference, Inverse Sensor Model based on an Inverse Detection Model

The reference we use in this work for an IDM-based radar ISM is based on the work of Loop et al. [12] and will be called Ray-ISM. First, we define the condition that an area in the scene at range $r$ and angle $\varphi$ is occupied as follows

[TABLE]

Using the Bayesian framework, the conditional probability of $\mathbf{s}_{o}^{(r,\varphi)}$ given range and angular measurements $\tilde{r},\tilde{\varphi}$ can be expanded to include the true range and angle of the measured object $R,\Phi$ as follows

[TABLE]

Here, $p(R|\tilde{r}),p(\Phi|\tilde{\varphi})$ are the range and angular noise models respectively. In this work, they are modeled as independent Gaussian noise. Eventually, $p(s_{o}^{(r,\varphi)}|R,\Phi)$ corresponds to the ideal sensor model which, given the true range and angle, does not depend on the measurements any more. In this work, we extend the ideal ISM from [12] to also consider the angle. To do so, we leave the ISM along the range independent of the angle and additionally model the angular component as being equal to the range ISM at $\varphi=\Phi$ and unknown everywhere else as shown in Fig. 3 (left).

Since the range component is independent of the angular, we can first integrate over $R$

[TABLE]

and arrive with $p(s_{o}^{(r,\varphi=\Phi)}|\Phi,\tilde{r})=p(s_{o}^{(r)}|\tilde{r})$ at the same model as proposed in [12]. The angular integration can then be formulated as

[TABLE]

Using the equalities

[TABLE]

the final inverse detection model (IDM) for our radar can be written as follows

[TABLE]

The ISM can then be obtained by accumulation the IDMs for each detection.

V-B Neural Network Models

As mentioned in sec. II-B, we are going to follow the work done in the domain of deep ISMs and use an U-Net architecture with skip connections similar to [20] as shown in Fig. 4. As activations, LeakyReLu are used in all layers expect for the last which is kept linear. All the neural network variants are trained using dropout to make them comparable. We distinguish between three network configurations based on their output activations.

The first one, which we will refer to as Soft-Net, uses the expectation of the dropout samples $\hat{\mathbf{y}}$ as an input for a softmax. The softmax activations are used in a cross-entropy loss to train against the belief targets. Hence it has an output dimension of $128\times 128\times 3$.

The second configuration, which we will refer to as Ev-Net, also uses $\hat{\mathbf{y}}$. However, in this case, the expectations are fed into the evidential framework as shown in Fig. 2 and hence only need two output channels, one for each evidence mass. The last configuration, which we will refer to as Ev-Net-S, is the same as Ev-Net with the difference that instead of using $\hat{\mathbf{y}}$ it uses $\hat{\mathbf{y}}_{s}$.

VI EXPERIMENTAL RESULTS

VI-A Scores

To evaluate the model performances we define the conditional probabilities of the estimates given free or occupied targets respectively in (17), (18). Additionally, we define a conflict state and its conditional probability in (19).

[TABLE]

The conflict is used as a robustness measure of the predictions. Moreover, we separate the scores into visible and hidden targets since their distributions are highly different. The discrimination between hidden and occluded areas is based on the LiDAR measurements.

VI-B Quantitative Results

The scores in Tab. II, II indicate that Ev-Net-S is too conservative since it reaches the smallest conflict and false rates while at the same time it fails to predict occupied space for both visible and hidden areas. In contrast, Soft-Net performs too aggressively by reaching the highest positive rates while also ending up with the highest conflict and false rates in all neural network approaches. Ev-Net, on the other hand, reaches small false and conflict rates while increasing the performance by a factor of $2.5$ in free and $5$ in occupied space prediction in visible areas relative to the Ray-ISM. In the hidden areas, Ev-Net still outperforms Ray-ISM, even though its performance is reduced.

VI-C Qualitative Results

These observations are also reflected in Fig. 5. Here, the rows correspond to different scenes, the first column corresponds to the radar input images, the next three columns to the Ray-ISM predictions for $\hat{b}_{f},\hat{b}_{o},\hat{u}$ followed by the predictions of Soft-Net, Ev-Net and Ev-Net-S. The moving objects are marked as red detections. The chosen scenes are two T-intersections (rows A $\&$ B) and a straight road with parked cars on each side (row C).

It can be seen that the neural network predictions are much denser than the Ray-ISM’s estimates. Moreover, the free and occupied predictions of the Soft-Net are more blurry than the evidential pendants. This is because the Soft-Net’s unknown state prediction more closely resembles estimates of hidden areas while the uncertainty about whether an area is occupied or not is distributed into the free and occupied belief masses. In contrast, the evidential networks learn to transfer those uncertainties into the unknown state which results in sharper free/occupied estimates. Moreover, the effect of learned aleatoric uncertainty can clearly be seen at the borders between free and occupied spaces. Here, the sensor noise results in noisy borders causing high unknown estimates. Additionally, the effect of epistemic uncertainty can be seen in areas with complex detection pattern like in the upper left of row B or in the areas behind the parked cars in row C. We also want to mention that moving objects (e.g. red detections at the bottom of the image in row A) do not result in occupied space.

VII CONCLUSION

In this work, we addressed the problem of describing the occupation state of an environment based on automotive radar measurements which provide sparse, noisy data. This problem is approached by employing the recently proposed evidential convolutional neural networks which can be used to infer a probability density function (PDF) over the classification result rather than a point prediction. Moreover, the uncertainty represented in the PDF can be used to reduce the classification probabilities accordingly by moving the mass to a separate unknown state. We have shown that this formulation provides an alternative to the typically applied detection-based inverse radar models by improving the performance by a factor of $2.5$ in detecting free and a factor of $5$ in detecting occupied space on our data set. At the same time, in contrast to typical softmax-based classification networks, the model remains robust to uncertainties and reaches low false and conflict rates. Moreover, we have shown how to utilize more spatial information by using occupancy map patches instead of the LiDAR-ISM as targets during training.