DeepLocalization: Landmark-based Self-Localization with Deep Neural Networks

TL;DR

DeepLocalization employs deep neural networks to accurately and efficiently determine vehicle pose from landmark data, demonstrating robustness to environmental changes and outperforming existing methods in speed and accuracy.

Contribution

This work introduces a novel neural network architecture for landmark-based vehicle self-localization that is robust, fast, and capable of integrating with GPS and filtering algorithms.

Findings

Achieves state-of-the-art localization accuracy.

Operates approximately ten times faster than previous methods.

Effectively handles dynamic environments with few landmarks.

Abstract

We address the problem of vehicle self-localization from multi-modal sensor information and a reference map. The map is generated off-line by extracting landmarks from the vehicle's field of view, while the measurements are collected similarly on the fly. Our goal is to determine the autonomous vehicle's pose from the landmark measurements and map landmarks. To learn this mapping, we propose DeepLocalization, a deep neural network that regresses the vehicle's translation and rotation parameters from unordered and dynamic input landmarks. The proposed network architecture is robust to changes of the dynamic environment and can cope with a small number of extracted landmarks. During the training process we rely on synthetically generated ground-truth. In our experiments, we evaluate two inference approaches in real-world scenarios. We show that DeepLocalization can be combined with regular GPS signals and filtering algorithms such as the extended Kalman filter. Our approach achieves state-of-the-art accuracy and is about ten times faster than the related work.

Figures

Tables

Table 1. TABLE I: Overview of results for synthetic measurements, real-world scenarios and related approaches.

	RMSE			inference
Experiment	$x$	$y$	$\phi$	time
Synthetic Measurements:
DeepLocalization
${\sigma }_{xy}=2m,{\sigma }_{\phi }={10}^{\circ }$	$0.178m$	$0.170m$	${0.852}^{\circ }$	$1.758\mathrm{ms}$
GPS-based Inference:
DeepLocalization
${\sigma }_{xy}=2m,{\sigma }_{\phi }={10}^{\circ }$	$0.773m$	$0.701m$	${2.52}^{\circ }$	$1.771\mathrm{ms}$
DeepLocalization
${\sigma }_{xy}=1m$, ${\sigma }_{\phi }=4^{\circ }$	$0.447m$	$0.377m$	${1.37}^{\circ }$	$1.761\mathrm{ms}$
DeepLocalization
${\sigma }_{xy}=0.5m$, ${\sigma }_{\phi }=2^{\circ }$	$0.276m$	$0.231m$	${0.85}^{\circ }$	$1.728\mathrm{ms}$
Filter-based Inference:
DeepLocalization
+ EKF	$0.566m$	$0.339m$	${1.26}^{\circ }$	$2.133\mathrm{ms}$
DeepLocalization
+ EKF + GPS	$0.271m$	$0.245m$	${0.82}^{\circ }$	$2.374\mathrm{ms}$
Related Approaches:
ICP [18]	$1.174m$	$1.467m$	${4.96}^{\circ }$	$7.601\mathrm{ms}$
EKF + GPS	$0.590m$	$0.547m$	${6.58}^{\circ }$	$0.168\mathrm{ms}$
RFS-MCL + GPS [11]	$0.287m$	$0.264m$	${1.99}^{\circ }$	$28.92\mathrm{ms}$

Equations

$$f_{\theta }\left({\left\{{\mathbf{z}}_1,\dots ,{\mathbf{z}}_{\nu }\right\}}_t,{\left\{{\mathbf{m}}_1,\dots ,{\mathbf{m}}_{\mu }\right\}}_{t-1}\right)\to \left[\Delta \hat{x},\Delta \hat{y},\Delta \hat{\phi }\right].$$

$${\mathbf{p}}_t={\mathbf{p}}_{t-1}+[\Delta \hat{x},\Delta \hat{y},\Delta \hat{\phi }{]}^T,$$

$${\mathrm{L}}_{\mathrm{tran}}=\mathbb{E}\left[(\Delta \hat{x}-\Delta x{)}^2\right]+\mathbb{E}\left[(\Delta \hat{y}-\Delta y{)}^2\right].$$

$${\mathrm{L}}_{\mathrm{rot}}=\mathbb{E}\left[(\Delta \hat{\phi }-\Delta \phi {)}^2\right].$$

$$\mathrm{L}={\mathrm{L}}_{\mathrm{tran}}{e}^{-s_{\mathrm{tran}}}+s_{\mathrm{tran}}+{\mathrm{L}}_{\mathrm{rot}}{e}^{-s_{\mathrm{rot}}}+s_{\mathrm{rot}},$$

$${\hat{\mathbf{p}}}_{\mathrm{UTM}}={\mathbf{p}}_{\mathrm{UTM}}+[\Delta x,\Delta y,\Delta \phi {]}_{{\sigma }_x,{\sigma }_y,{\sigma }_{\phi }}^T,$$

$$H=\left[\begin{array}{ccc}\cos ({\hat{\phi }}_{\mathrm{UTM}})& -\sin ({\hat{\phi }}_{\mathrm{UTM}})& {\hat{x}}_{\mathrm{UTM}}\\ \sin ({\hat{\phi }}_{\mathrm{UTM}})& \cos ({\hat{\phi }}_{\mathrm{UTM}})& {\hat{y}}_{\mathrm{UTM}}\\ 0& 0& 1\end{array}\right].$$

$${\mathbf{p}}_t={\mathbf{p}}_{\mathrm{GPS}}+[\Delta \hat{x},\Delta \hat{y},\Delta \hat{\phi }{]}^T,$$

$${\mathbf{p}}_t^{\mathrm{EKF}}=\mathrm{EKF}({\mathbf{p}}_t),$$

Full text

© 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

I Introduction

Self-localization is the task of determining the vehicle’s position and orientation (pose) within a coordinate system, based on sensor information [1]. In autonomous driving, the precise vehicle position is essential for path planning [2]. Furthermore, it increases the vehicle’s safety and reliability by incorporating prior information from a digital map for the environment perception [3].

The required localization accuracy is, of course, subject to the driving environment. In structured and organized environments, e.g. freeways, the localization accuracy can be as low as a couple of meters under certain conditions. Regular global navigation satellite systems (GNSS), such as GPS, usually provide sufficient results for these tasks [4]. However, more complex and unstructured scenarios in urban and rural areas demand an extensive environment perception with a precise localization, requiring accuracies in the range of 0.2\text{\,}\mathrm{m}$-$0.5\text{\,}\mathrm{m}. Achieving this precision is possible with a dGPS-System that relies on correction data from a network of ground-based reference stations. However, this is not a sustainable solution for every autonomous driving vehicle due to the high cost of the system. In addition, all GNSS systems heavily rely on satellite coverage, which can be poor in cities where the clear sight to the satellites is concealed by tall buildings [5]. The most adopted solution in localization is to combine multiple sensors, e.g. radar, camera, lidar and GPS, with a high-precision map to estimate the vehicle’s pose [6, 7, 8].

Classical probabilistic localization approaches have shown great results in unstructured environments enabling the successful deployment of autonomous vehicles in many cities around the world using either a grid-based representation of the environment [9] or a landmark-based map. In the latter case, the map contains static and easily recognizable objects [10, 11]. Storing objects in a landmarks-based map requires small amount of memory compared to the grid-based maps, where the area is discretized into small cells. Moreover, landmark-based maps are easier to maintain and update, because landmarks can be added or removed without much effort.

In this work, we present an approach for self-localization based on landmarks. By relying on multi-modal sensor information, we extract a set of landmarks from the vehicle’s field of view and build a map off-line. During localization, the autonomous vehicle extracts landmarks on the fly that are matched with the map landmarks and, eventually, estimates the vehicle’s pose. Note that there are map landmarks which will not be visible after building the map due to the dynamic environment (e.g different lighting and weather conditions, dynamic objects and infrastructure). Furthermore, estimating the vehicle’s absolute pose in a global coordinate system is highly ambiguous and thus we rely on the vehicle’s pose from the previous time step to estimate the current pose. On top of the environment challenges, we also have to deal with input of dynamic size since the number of landmarks in the vehicle’s field of view varies over time. As a result, matching the measured landmarks that are extracted in real-time with the map landmarks is not straightforward. To address these issues, we introduce a deep neural network that learns to match the measured landmarks with the map landmarks based on synthetically generated ground-truth.

Traditional convolutional neural networks are not well-suited for our problem, because they require the input data to be structured. Moreover, transforming the dynamic input into an ordered representation is not trivial. We derive our motivation from the latest network architectures for unstructured 3D data processing. The recently introduced PointNet architecture [12] copes with unordered point lists. The main idea of PointNet is to create a signature of the dynamic input list, i.e. the 3D point cloud, by combining a multi-layer perceptron (MLP) with the max-pooling operation.

We propose a neural network architecture that receives input from two dynamic lists to regress the vehicle’s pose. The measured landmarks compose one dynamic list and the map landmarks the other one. Our architecture does not explicitly model the matching process. Instead, it is a vehicle’s pose regressor with measurements and map landmarks as input. Moreover, the pose is not expressed in the global coordinate system as discussed above. Our networks regress the vehicle’s pose w.r.t previous pose, i.e. a pose offset. During inference, our only requirement is the availability of at least a single GPS measurement to initiate our pipeline. We present two inference algorithms to show that our approach improves the GPS measurements and is also compatible with filtering algorithms, such as the extended Kalman filter.

In our experiments, we study the performance of our approach on our own dataset which was recorded in Ulm-Lehr, Germany. First, we perform a number of synthetic experiments to show the generalization capabilities of our network. Then, we compare our results with prior work to show better localization accuracy and faster inference times in the range of $2\,$\mathrm{ms}$$. We call our approach DeepLocalization. Some of our results are shown in Fig. 1.

II Related Work

Self-Localization is a fundamental problem in robotics. Usually, the standard scenario is to estimate the robot’s pose by accumulating noisy sensor measurements using filtering algorithms such as the extended Kalman filter (EKF) or particle filter, i.e. Monte-Carlo localization (MCL) [1, 13, 14]. Later, self-localization has been raised to a vital task for autonomous driving vehicles too. Several approaches have been proposed in the past few years. Below, we discuss the methods that are related to ours as well as deep learning approaches.

Classic Self-Localization. A common way of categorizing the localization methods is based on the type of environment representation. The majority of the approaches rely on grid-based maps, i.e. a discretization of the environment into 2D or 3D cells. In that case, features of the environment are generated for every cell and saved in an off-line map. In the on-line phase, the same feature extraction algorithms are used for producing measurements. By aligning the off-line map with the measurements the vehicle’s pose can be estimated. Levinson et al. [15, 16] proposed a probabilistic grid, where the remittance value of lidar measurements is modeled for each cell as its own Gaussian distribution. Wolcott et al. [17] proposed the Gaussian mixture maps in which each cell of the 2D grid contains Gaussian mixture models over the height (vertical structure), measured by multiple lidar scanners. The advantage over approaches which use the reflectivity of lidar scans, e.g. [16], lies in the robust localization accuracy even in adverse weather conditions. Besides the grid-based environment representation, landmark-based maps are a popular choice in autonomous driving applications too. The maps usually contain sets of independent 2D or 3D points to represent distinct and recognizable objects. Traditionally, observed point clouds from different sensors are registered to the pre-built map using the iterative closest point algorithm [18] or similar variations [19]. Instead of searching for a single best solution for the registration problem, several approaches use Monte Carlo localization methods in combination with a digital map and sensor measurements to determine the vehicle’s pose [11, 20].

Deep Localization. Deep learning approaches have shown promising results in regression task [21], including localization. Kendall et al. proposed PoseNet [22], a convolutional neural network that regresses the camera’s $6$ degrees of freedom pose relative to the scene from a single RGB image. A similar approach has been introduced by Valada et al. with VLocNet [23]. The goal is to regress the global pose and simultaneously estimate the odometry between two frames. The network is based on residual neural networks that take two consecutive monocular images as input. DeepMapping [24] aligns point clouds to a global coordinate system, which is closer to our approach. The method regresses the sensor’s pose using a deep neural network followed by a mapping network that models the structure of a scene using grid maps. Contrary to this method, our model directly learns the alignment transformation from the two dynamic inputs without requiring an intermediate step.

Finally, it is important to note that we derive inspiration from PointNet [12] to process unstructured data such as point clouds. PointNet uses sets of 3D points to learn a global feature representation of the input which can be used for classification and segmentation. Although PointNet is not suited for processing input from multiple sources, it is related to our model. We propose an architecture within the context of localization that regresses the vehicle’s pose from two dynamic input lists, the measurements and map landmarks.

III Method

We consider multi-modal measurements from lidar, radar and camera sensors. For each time step, a set of landmarks is extracted from the raw data using pre-processing algorithms [11]. All extracted landmarks from the vehicle’s field of view are registered to the same coordinate system. This is performed off-line and constitutes our map building process. In general, our map is comprised of 2D points, each resembling a static and recognizable object [1]. In the on-line phase, landmarks, which we refer to as measurements, are extracted the same way. The goal is to localize the vehicle w.r.t to the map landmarks by utilizing the measurements and the vehicle’s initial position.

It is important to mention that there exists no specific order in which the measurements are obtained. Furthermore, the number of landmarks is not known in advance and it varies over time. We present an approach that copes with dynamic and unordered input.

III-A Problem Definition

We seek for mapping the measurements $\{\mathbf{z}_{1},\dots,\mathbf{z}_{\nu}\}$, $\mathbf{z}_{\{\cdot\}}\in\mathbb{R}^{2}$ to the map landmarks $\{\mathbf{m}_{1},\dots,\mathbf{m}_{\mu}\}$, $\mathbf{m}_{\{\cdot\}}\in\mathbb{R}^{2}$ to obtain the vehicle’s pose $[x,y,\varphi]$. In our representation, the measurements and map landmarks are comprised of 2D points. Regressing directly the absolute position in a global coordinate system, such as the Universal Transverse Mercator (UTM) coordinate system, is highly ambiguous. Instead, we propose to infer the pose offset between the previous and current time step. We make use of the vehicle’s pose from the previous (or initial) time step $t-1$ and the current measurements to predict the pose offset $[\Delta x,\Delta y,\Delta\varphi]$ to the current pose. The position offset $\Delta x$ and $\Delta y$ can be interpreted as translation and the orientation offset $\Delta\varphi$ as rotation.

We present a deep neural network architecture that receives the measurements of the current time step $t$ and the map landmarks from $t-1$ as input to predict the pose offset between the previous and current time step. The idea is to learn the relation between measurements and map landmarks. The proposed mapping is parameterized by $\theta$ and is given by:

[TABLE]

Note that we have 2-axis of translation and 1-axis of rotation in our problem, which is a common way of describing an agent’s pose [1]. Based on the output of $f_{\theta}$ and the previous vehicle’s pose $\mathbf{p}_{t-1}$, we can obtain the current pose $\mathbf{p}_{t}$. This is described by:

[TABLE]

where $\mathbf{p}_{t},\mathbf{p}_{t-1}\in\mathbb{R}^{3}$.

III-B Objective

The learning objective is to minimize the difference between the predicted $[\Delta\hat{x},\Delta\hat{y},\Delta\hat{\varphi}]$ and ground-truth pose offsets $[\Delta x,\Delta y,\Delta\varphi]$. We define the translation and rotation loss terms that are treated independently. The reason is the different magnitude of the units, i.e. meters and degree, that require scaling the two terms for equal gradient contribution. At first, the translation loss is given by:

[TABLE]

Secondly, the rotation loss is defined as:

[TABLE]

We found that manually tuning the weighting of the two terms is a tedious task. Furthermore, the weights can be sensitive to small modifications in the training data. Rather than choosing a data specific weighting strategy, we learn the weighting factors [25]. The complete loss function is defined as:

[TABLE]

where $s_{\mathrm{tran}}=\log\sigma_{\mathrm{tran}}^{2}$ and $s_{\mathrm{rot}}=\log\sigma_{\mathrm{rot}}^{2}$. The terms $\sigma_{\mathrm{tran}}$ and $\sigma_{\mathrm{rot}}$ denote the homoscedastic uncertainty, derived from Bayesian modeling [26]. Homoscedastic uncertainty is a sub type of aleatoric uncertainty that is not dependent on the input data. It is therefore constant but varies between different tasks. Following [25], homoscedastic uncertainty can be interpreted as task-dependent weighting, which can be learned by the network. In our formulation, the tasks correspond to the translation and rotation. Our loss function is similar to the geometric loss function in [27], where the only differences are the degrees of freedom.

III-C Network Architecture

DeepLocalization, our network architecture, is presented in Fig. 2. The inputs of DeepLocalization are two independent point lists, namely the measurements and map landmarks. For each of the two input lists we employ a separate MLP with three layers, denoted by $\text{MLP}_{\text{meas}}$ and $\text{MLP}_{\text{map}}$ in Fig. 2, which transform the input to a higher dimensional representation $\mathbb{R}^{D}$, where $D=1024$. Note that the weights of $\text{MLP}_{\text{meas}}$ are shared for every measurement, similar for the map landmarks and $\text{MLP}_{\text{map}}$. The output of each MLP is $\nu\times 1024$ for the measurements and $\mu\times 1024$ for the map landmarks. At the end of $\text{MLP}_{\text{meas}}$ and $\text{MLP}_{\text{map}}$, a max-pooling operation follows to produce the global feature vector of $D$ elements. One can observe that the global feature vector does not depend on the number of input landmarks which makes the network invariant to the input size. In addition, the global feature vector is invariant to input permutations [12]. Finally, the two global feature vectors are concatenated and fed to a third $\text{MLP}_{\text{offset}}$. The role of $\text{MLP}_{\text{offset}}$ is to find the correlation between the measurements and map landmarks, which now have been projected to the same dimensions. The output of $\text{MLP}_{\text{offset}}$ is the vehicle’s pose offset.

It’s important to note that the map landmarks are loaded only for a radius of $100\text{\,}\mathrm{m}$ around the vehicle’s noisy position. During training, we do have the ground-truth information for the vehicle’s location and, consequently we can load the corresponding map landmarks. During testing, we make use of the previous or initial vehicle’s location to load the map landmarks. Although, the radius around the vehicle from which the landmarks are loaded is slightly delayed in time, we have found that it does not have any negative impact on the pose offset prediction.

III-D Training Process

The training of our model is performed with synthetic pose offsets. We randomly fetch from the training data the following: the vehicle’s ground-truth pose in the UTM coordinate system $\mathbf{p}_{\mathrm{UTM}}=[x_{\mathrm{UTM}},y_{\mathrm{UTM}},\varphi_{\mathrm{UTM}}]$, the map landmarks $\{\mathbf{m}_{1},\dots,\mathbf{m}_{\mu}\}$ in the UTM coordinate system and the measurements $\{\mathbf{z}_{1},\dots,\mathbf{z}_{\nu}\}$ in the vehicle coordinate system. In addition, we randomly sample a pose offset $[\Delta x,\Delta y,\Delta\varphi]_{\sigma_{x},\sigma_{y},\sigma_{\varphi}}$ from a uniform distribution on the interval $\Delta x\in[-\sigma_{x},\sigma_{x}]\,$\mathrm{m}, $\Delta y\in[-\sigma_{y},\sigma_{y}]\,$\mathrm{m} and $\Delta\varphi\in[-\sigma_{\varphi},\sigma_{\varphi}]^{\circ}$. The pose offset composes the desired network output, i.e. ground-truth, and is applied to the vehicle’s pose at the UTM coordinate system such that it causes a shift. This is written as:

[TABLE]

where $\hat{\mathbf{p}}_{\mathrm{UTM}}=[\hat{x}_{\mathrm{UTM}},\hat{y}_{\mathrm{UTM}},\hat{\varphi}_{\mathrm{UTM}}]$. Based on the shifted vehicle’s pose in the UTM coordinate system, we define the homogeneous transformation matrix $H$ that transforms a measurement from the vehicle to the UTM coordinate system. This is given by:

[TABLE]

We can use $H$ to transform all measurements to the UTM coordinate system. However, we perform the opposite by transforming the map landmarks to the vehicle coordinate system with the inverse transformation matrix $H^{-1}$ for numerical stability reasons. The range of values in the vehicle coordinate system is better suited for training a neural network. Moreover, we predict offsets, i.e. translation and rotation, which are independent of the coordinate system. At this point, the map landmarks and the measurements are in the vehicle coordinate system.

This procedure allows us, theoretically, to generate an infinite amount of training data. Moreover, the sampling order over time does not play any role in our optimization. Thus, we can sample training data at random time steps.

IV Inference

We present two inference approaches. First, the GPS-based inference is based on improving the GPS sensor measurement using our approach at every time step. Second, the filter-based inference combines our approach with an extended Kalman filter (EKF). The idea is to make use of the temporal information.

IV-A GPS-based Inference

The first inference algorithm relies on the GPS measurement for obtaining an initial but noisy vehicle pose. Here, we do not make use of the pose from the previous time step. Thus, we rewrite Eq. (2) as:

[TABLE]

where $\mathbf{p}_{\mathrm{GPS}}$ is the noisy GPS vehicle pose in the UTM coordinate system. To obtain the pose offsets $\Delta\hat{x}$, $\Delta\hat{y}$ and $\Delta\hat{\varphi}$ from DeepLocalization, we follow the same pipeline as in the training process from Sec. III-D. The loaded map landmarks are transformed to the vehicle coordinate system based on $H^{-1}$, where the position and orientation from $\mathbf{p}_{\mathrm{GPS}}$ are used in $H$. The same process is followed at every time step to estimate $\mathbf{p}_{t}$.

IV-B Filter-based Inference

The filter-based algorithm requires an initial position, which can be retrieved from the GPS sensor. We rely on Eq. (2) to predict the vehicle’s position in the UTM coordinate system. At time step $t$, the measurements become available in the vehicle coordinate system. Moreover, the map landmarks are loaded based on the previous pose $\mathbf{p}_{t-1}$. The map landmarks are expressed in the vehicle coordinate system by applying the transformation $H^{-1}$, which is computed from the vehicle’s previous pose $\mathbf{p}_{t-1}$, since $\mathbf{p}_{t}$ is not available. The measurements and map landmarks in the vehicle coordinate system are provided to DeepLocalization to regress the pose offset $[\Delta\hat{x},\Delta\hat{y},\Delta\hat{\varphi}]$. Finally, the new pose $\mathbf{p}_{t}$ is calculated by applying the estimated offset to $\mathbf{p}_{t-1}$.

Nevertheless, our network does not take into consideration the time domain, which could help to avoid pose drifts. To address this issue, we additionally propose to combine our network with an EKF that has a constant turn rate and velocity (CTRV) motion model. The EKF takes the new vehicle pose $\mathbf{p}_{t}$, i.e. Eq. (2), and refines it such that

[TABLE]

where $\mathrm{EKF}$ represents the filter. At the next time step, the pose $\mathbf{p}_{t}^{\mathrm{EKF}}$ is used as $\mathbf{p}_{t-1}$ and the process is repeated with new measurements. An overview of the filter-based inference method is shown in Fig. 3.

V Ulm-Lehr Dataset and Map Creation

We introduce our dataset to examine DeepLocalization in real-world scenarios. To that end, we have made recordings with our vehicle, equipped with a stereo camera setup, lidar and radar sensors, as well as a dGPS system for generating the ground-truth pose. Our test track in Ulm-Lehr, Germany, which is about $5\,$\mathrm{km}$$ long, can be seen in Fig. 4 along with the landmarks from the map.

The landmark-based map has been created in December 2017 by recording three runs on different days at our test track. It consists of $3860$ landmarks, of which $1731$ were obtained from lidar measurements, $1411$ from camera images and $718$ from radar targets. Independently, the measurement dataset has been built from eight runs on the Ulm-Lehr route in November 2018. In total, the dataset includes approximately $145.000$ samples, each sample containing measurements and the ground-truth pose. Six runs ($105.000$ samples) are used for training and validation. The remaining two runs ($40.000$ samples) are used for test. Below, we discuss the landmark extraction process and the generation of the landmark-based map in detail.

V-A Landmark Extraction

All measurement recordings take place with the same sensor configuration. While driving the route, we record measurements in the form of 3D point clouds for the lidar and radar sensors and RGB images with depth (RGB-D) from the stereo system. We extract features because we found them to be more stable and computational efficient than raw measurements. For the lidar and radar measurements, the density based DBSCAN [28] cluster algorithm is applied to group adjacent points that have a minimum amount of neighboring points within a predefined distance. For the RGB-D images, maximally stable extremal regions (MSER) are extracted [29]. The MSER algorithm is parameterized to detect road markings since they are permanent and thus can be detected recurrently. All features are stored in the vehicle coordinate system that has the origin at the center of the rear axle. The image features, in particular, are transformed to the vehicle coordinate system using the depth map. The extracted features from lidar, radar and RGB-D images define what we call landmarks. Additionally, the dGPS system records the ground-truth pose of the vehicle. It consists of the heading (yaw) and latitude and longitude coordinates that are transformed into Cartesian coordinates (UTM).

V-B Landmark-based Map

The landmark extraction process (Sec. V-A) is sufficient for extracting landmarks as measurements, but it is not enough for building our map. There are several landmark instances that are repeated several times. To avoid storing the same landmark multiple times, we additionally rely on the LMB-SLAM algorithm [10] to make the landmark-based map sparser. In addition, dynamic objects, such as vehicles or time-dependent structures, may produce landmarks which are not valid for future use. To discard this type of landmarks, we fuse all three runs from December 2017 with a Bernoulli filtering approach [30]. Finally, the map landmarks are expressed in the UTM coordinate system. We have transformed the landmarks from the vehicle to the UTM coordinate system by incorporating the dGPS measurements. Therefore, the map landmarks have a relative accuracy of a few centimeters. In total, the map has a size of $600\,$\mathrm{kB}$$.

VI Experiments

We conduct two experiments on the Ulm-Lehr dataset to evaluate our approach. First, we examine our model with synthetic measurements and pose offsets. Second, we evaluate on the test set and compare our results with related approaches. For all experiments, we rely on the same DeepLocalization model that has about $1.8$ million learnable parameters. Our model has been trained with the ADAM optimizer [31] and a learning rate of $10^{-5}$. The batch size was set to $500$ samples. Additionally, all MLP networks have dropout [32] after every layer as regularizer, except the last one. During test, the inference time of our network is $1.758\,$\mathrm{ms}$$. Note for all evaluated approaches, including ours, the feature extraction phase is not included in the inference time. Our approach is implemented in TensorFlow [33] and all operations are performed on a GPU with 11GB memory.

VI-A Synthetic Measurements Evaluation

We use synthetic measurements to examine the lower error bound of our approach. We also access how DeepLocalization generalizes under error-prone, missing or additional measurements.

For each time step, the map landmarks in the vehicle’s field of view are used as measurements too. When the measurements and map landmarks are identical, the input is ideal and the only source of error can be the incorrect offset prediction of the network. To that end, we transform the measurements and map landmarks to the vehicle coordinate system. Then, a synthetic vehicle pose offset is added to the map landmarks as in the training process (Eq. (6)). To generate the vehicle pose offset, we sample noise from a uniform distribution on the interval $\sigma_{x},\sigma_{y}\in[-2,2]\,$\mathrm{m} and $\sigma_{\varphi}\in[-10,10]\,^{\circ}$, which are realistic accuracies for modern GPS receivers [[34](#bib.bib34)]. We train our model based on the synthetic pose offsets. During test, our network receives the measurements and shifted map landmarks to predict the pose offset. Since the ground-truth is the synthetic vehicle pose offset, we can measure the prediction accuracy with the RMSE metric. We generate around $40.000$ test samples and achieve a RMSE of $0.178\,$\mathrm{m} and $0.17\,$\mathrm{m}$$ for the $x$ and $y$ coordinates, respectively. Furthermore, the orientation error is $0.852\,^{\circ}$. The results are also summarized in Tab. I.

Next, we consider the following situations:

Randomly delete measurements (simulating missed landmarks). 2. 2.

Randomly add measurements (simulating clutter). 3. 3.

Add randomly sampled noise to each measurement (simulating sensor noise). 4. 4.

Add all three impairments simultaneously.

In practice, the rates at which measurements are added and deleted follow a Poisson distribution $P_{\lambda}(k)=\lambda^{k}\exp(-\lambda)/(k!)$, where $\lambda$ is the average number of events. The measurement noise for the third experiment is sampled from a uniform distribution on the interval $[-\sigma_{syn},\sigma_{syn}]$. The evaluations are performed by increasing $\lambda$ and $\sigma_{syn}$. Our results are presented in Fig. 5.

One can observe that with clutter rates of $\lambda_{clutter}=40$, the error remains low at $0.4\,$\mathrm{m}, even though the average number of actual measurements is about $20-30$ per time step. Similar results are observed for missed detections with $\lambda_{miss}=10$ and uniformly sampled noise for every landmark of about $\sigma_{syn}\in[-0.9,0.9]\,$\mathrm{m}. Finally, the robustness of our approach is demonstrated when all three impairments occur simultaneously. For instance, in iteration $10$ ($\lambda_{clutter}=\lambda_{miss}=10$ and $\sigma_{syn}=0.27\,$\mathrm{m}) the RMSE value for $x$ and $y$ is about $0.5\,$\mathrm{m} and for the orientation $1.87^{\circ}$. When the effects become too severe (e.g. in iteration $18$), the measurements and the map landmarks diverge and, consequently, the localization fails.

VI-B Ulm-Lehr Evaluation

For this experiment, we train our network with the measurements and map landmarks from the Ulm-Lehr train set. To examine the robustness of our approach, we generate synthetic noise offsets to train three network models with different noise parameters. For the first setup, we sampled noise from a uniform distribution on the interval $\sigma_{x}\in[-2,2]\,$\mathrm{m}, $\sigma_{y}\in[-2,2]\,$\mathrm{m} and $\sigma_{\varphi}\in[-10,10]\,^{\circ}$. For the second case, the noise is sampled on the interval $\sigma_{x}\in[-1,1]\,$\mathrm{m}, $\sigma_{y}\in[-1,1]\,$\mathrm{m} and $\sigma_{\varphi}\in[-4,4]\,^{\circ}$; and the sampled noise for the last case was on the interval $\sigma_{x}\in[-0.5,0.5]\,$\mathrm{m}, $\sigma_{y}\in[-0.5,0.5]\,$\mathrm{m} and $\sigma_{\varphi}\in[-2,2]\,^{\circ}$. In all three experiments we use GPS-based inference in order to improve the localization over the noisy GPS measurement. The results are summarized in Tab. I.

Next, we evaluate the filter-based inference using the EKF. Here, we have relied on a single GPS measurement for initialization. Afterwards, the algorithm uses only the network output to obtain the current position, i.e. our prediction offset updates the pose from time step $t-1$ to $t$. Thus, after initialization, we are independent of any GPS measurements. After evaluating the three models, we have concluded that the network with noise parameters $\sigma_{x}=1\,$\mathrm{m}, $\sigma_{y}=1\,$\mathrm{m}, $\sigma_{\varphi}=4\,^{\circ}$ performs at best when combined with the EKF with a RMSE of $0.566\,$\mathrm{m} and $0.339\,$\mathrm{m} for the $x$ and $y$ position, respectively. Thus, we only report this result in Tab. I for the EKF evaluation. The EKF is updated every $20~{}$\mathrm{ms}$$ and the measurement noise matrix was obtained empirically.

In addition, we compare DeepLocalization with three approaches. We evaluate the iterative closest point algorithm (ICP) [18], an extended Kalman Filter with only noisy GPS measurements as input and a random finite set Monte Carlo localization method (RFS-MCL) [11].

The ICP algorithm iteratively tries to minimize the difference between two point clouds, where one point cloud is fixed and the other one is transformed to match the reference point cloud. As initial guess, we transform the landmarks into the vehicle coordinate system using $H^{-1}$ with a noisy GPS measurement. Finally, the map landmarks are aligned to the measurements. Since the ICP algorithm is computationally expensive, the inference time is $7.601~{}$\mathrm{ms} with accuracies (RMSE) of $1.1~{}$\mathrm{m}$-1.4~{}$\mathrm{m} for the $x$ and $y$ coordinates and $5^{\circ}$ for the orientation.

The EKF with only noisy GPS measurements as input achieves slightly better results for the position accuracy which are reported in Tab. I. The Monte-Carlo localization method uses the same map and features from the multi-modal measurements. Unlike our learning-based approach, Stuebler et al. model the measurements and landmarks as Random Finite Sets (RFS), and estimate the vehicle’s pose with a particle filter. The weights of the particles are updated using a Bernoulli filtering approach. An in-depth explanation of the RFS-MCL method can be found in [11]. We have re-implemented the approach with a total of $1000$ particles which translates to $28.92\,$\mathrm{ms} of inference time. In our implementation, the method uses additional GPS measurements. For that reason, we also evaluate results by incorporating GPS measurements in the EKF filter, i.e. DeepLocalization + EKF + GPS in Tab. [I](#S6.T1). We achieve position RMSE in the range of $0.245\,$\mathrm{m}$-0.271\,$\mathrm{m} and an orientation error of $0.82^{\circ}$. Our approach has a computational time of $2.374\,$\mathrm{ms}$$, which is about ten times faster than the traditional RFS-MCL. Finally, a visual illustration of DeepLocalization with an EKF is shown in Fig. 6. It becomes apparent that our prediction is very close to the ground-truth position in the UTM coordinate system.

VII Conclusion

In this work, we have proposed a new approach for vehicle self-localization. We have introduced DeepLocalization, a deep neural network, that regresses the translation and rotation offsets from unordered and dynamic input lists. We have presented two inference algorithms, namely GPS-based and filter-based inference. In the evaluation, we have shown better performance than the prior work on the Ulm-Lehr dataset. In addition, our approach runs ten times faster than the RFS-MCL. As future work, we would like to explore how to be completely independent of GPS measurements.