Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR Point Cloud Data

TL;DR

This paper introduces a domain adaptation framework using CycleGAN to improve vehicle detection accuracy in real LiDAR point cloud data by reducing the domain gap with synthetic data.

Contribution

It proposes a novel CycleGAN-based domain adaptation method specifically designed for vehicle detection in bird's eye view LiDAR point clouds.

Findings

Over 7% improvement in average precision on real BEV point cloud data

Effective bridging of synthetic and real LiDAR data domains

Enhanced vehicle detection performance in safety-critical applications

Abstract

Point cloud data from 3D LiDAR sensors are one of the most crucial sensor modalities for versatile safety-critical applications such as self-driving vehicles. Since the annotations of point cloud data is an expensive and time-consuming process, therefore recently the utilisation of simulated environments and 3D LiDAR sensors for this task started to get some popularity. With simulated sensors and environments, the process for obtaining an annotated synthetic point cloud data became much easier. However, the generated synthetic point cloud data are still missing the artefacts usually exist in point cloud data from real 3D LiDAR sensors. As a result, the performance of the trained models on this data for perception tasks when tested on real point cloud data is degraded due to the domain shift between simulated and real environments. Thus, in this work, we are proposing a domain adaptation framework for bridging this gap between synthetic and real point cloud data. Our proposed framework is based on the deep cycle-consistent generative adversarial networks (CycleGAN) architecture. We have evaluated the performance of our proposed framework on the task of vehicle detection from a bird's eye view (BEV) point cloud images coming from real 3D LiDAR sensors. The framework has shown competitive results with an improvement of more than 7% in average precision score over other baseline approaches when tested on real BEV point cloud images.

Figures

Tables

Table 1. Table I: Comparison between our 5 trained YOLOv3 models on the same testing split BEV point cloud images from the KITTI dataset  [ 11 ] . Higher is better.

Model	Training Data	Average Precision (AP)%
$YOL{O}_S$	SYN (only)	29.93
$YOL{O}_R$	DA (only)	34.78
$YOL{O}_K$	KITTI (only)	57.26
$YOL{O}_{KS}$	KITTI+SYN	59.16
$YOL{O}_{KR}$	KITTI+DA	64.29

Equations

$$\begin{array}{r}{\mathcal{L}}_{ad{v}_{S\to R}}=\underset{G_{S\to R}}{\min }\underset{D_R}{\max }\underset{r\sim {\mathbb{P}}_d(r)}{\mathbb{E}}[\log D_R(r)]+\\ \underset{s\sim {\mathbb{P}}_d(s)}{\mathbb{E}}[\log (1-D_R(G_{S\to R}(s)))]\end{array}$$

$$\begin{array}{r}{\mathcal{L}}_{ad{v}_{R\to S}}=\underset{G_{R\to S}}{\min }\underset{D_S}{\max }\underset{s\sim {\mathbb{P}}_d(s)}{\mathbb{E}}[\log D_S(s)]+\\ \underset{r\sim {\mathbb{P}}_d(r)}{\mathbb{E}}[\log (1-D_S(G_{R\to S}(r)))]\end{array}$$

$${\mathcal{L}}_{cyc}$$

$$+{\Vert G_{S\to R}\left(G_{R\to S}(r)\right)-r\Vert }_1$$

$$\mathcal{L}={\mathcal{L}}_{ad{v}_{S\to R}}+{\mathcal{L}}_{ad{v}_{R\to S}}+\lambda {\mathcal{L}}_{cyc}$$

Full text

I Introduction

Recently, deep learning-based techniques such as convolution neural networks (ConvNets) have been achieving state-of-the-art results in many computer vision tasks such: object identification [1], scene understanding [2, 3], and human action recognition [4, 5, 6]. However, these techniques require a handful amount of labelled data for training them which is both time-consuming and cumbersome to get for many tasks. Thus, the utilisation of synthetic data for training such techniques got some momentum over the past few years [7, 8]. With synthetic data, the process for obtaining ground-truth labels becomes much easier and automated most of the time. However, still, the utilisation of synthetic data is not entirely reliable because of its limitations when it comes to the generalisation to real data.

In safety-critical applications such as a self-driving vehicle, one of the main sensors that are currently crucial for its development is the 3D LiDAR (Light Detection And Ranging) sensor. 3D LiDAR sensors can reliably provide 360$\degree$ point cloud in traffic environment with coverage distance up to 200 meters ahead across different weather and lighting conditions. Thus, a number of deep-learning based techniques have recently been utilising its point cloud for many perception tasks for self-driving vehicles [9, 10]. The reason that the number of deep-learning techniques that rely on point-cloud data is not as much as the ones rely on visual data is the scarcity of labelled point cloud data. The labelling procedure for point cloud data is more complicated than visual data especially for tasks such as 3D object detection and per-point semantic segmentation. Thus, the usage of synthetic data has been explored, similar to the visual data modality data [7, 9]. However, the generalisation to real-point cloud data was rather limited due to the perfectness of the synthetic point cloud data (shown in Fig. 1, right) which is missing the artefacts usually exist in point cloud data from real 3D LiDAR sensors (shown in Fig. 1, left). These artefacts are such as the variability of the LiDAR beams intensities or the motion distortion as a result of the motion of the 3D LiDAR.

Domain adaptation (DA) is one of the machine learning (ML) techniques that have been recently explored to bridge the aforementioned gaps between synthetic and real data domains [12]. In DA, the goal is to learn from one data distribution (referred to as the source domain) a perfect model on a different data distribution (referred to as the target domain). In traffic environments, DA has recently shown promising results for image translation between different domain pairs such as night/day, synthetic/real images and RGB/thermal images [13]. Since most of the previous DA techniques are based on 2D deep ConvNet architectures, thus their application on 3D point cloud data from 3D LiDAR sensors is not a straight forward task.

On the other hand, the recent deep-learning based techniques that have been applied on perception tasks using 3D point cloud data, they managed to find a way to adopt the same 2D ConvNet architectures to work on the 3D point cloud data. One of the most common techniques was to project a top-down bird’s eye view (BEV) of the point cloud data on a 2D plane (ie. ground). The representation of the 3D LiDAR point cloud data as a BEV was shown to be effective in many perception tasks for self-driving vehicles such as 3D object detection [14], road detection [15] and per-point semantic segmentation [16].

To this end, in this work, we will be proposing a DA approach for vehicle detection in real point cloud data from 3D LiDAR sensors represented as BEV images. The proposed DA approach will be a deep learning-based approach based on deep generative adversarial networks (GANs) [13]. For the vehicle detection task, it will be based on state-of-the-art deep object detection architecture YOLOv3 [1]. The rest of the paper is organised as follows. In Section II, a brief introduction to the different DA approaches with emphasis on deep learning based approaches will be reviewed in addition to a quick review on GANs. Section III, the methodology we followed for our proposed DA approach will be discussed thoroughly. Experiments and results are discussed in Section IV. Finally, Section V concludes.

II Related Work

Commonly, there are two ways to achieve DA either by directly translating one domain to the other or by obtaining a common-ground intermediate pseudo-domain between the two domains. In the following, firstly a quick review of the work related to the DA approaches will be provided specifically the approaches based on the direct translation between domains. Then, a brief summary of the DA work between simulated and real domains done in the context of traffic environments will be discussed.

II-A Adversarial Domain Adaptation

Historically, most of the work done on DA has been relying on the transformation between source and target domains based on linear representations [17, 18]. Until the emergence of the recent set of techniques based on non-linear transformation representations via neural networks [19, 20], which have achieved state-of-the-art results in a number of DA benchmarks [21, 22]. One of the most commonly non-linear-based representations DA approaches is the adversarial domain adaptation (ADA) approach [19]. ADA was inspired by the work done by Goodfellow et al. [23] on generative adversarial networks (GANs). In GANs, there are two deep neural networks trained simultaneously, namely a “generator” network and a “discriminator” network. The generator network, as the name implies, it generates new data instances using a uniform distribution, on the other hand, the discriminator network tries to decide whether or not this newly generated data instance has the same distribution as the training dataset distribution. Similarly, in ADA, it has the same two networks, where the generator network, generates instances from the source domain distribution to transform it into the target domain distribution. Whereas, the discriminator network tries to differentiate between the instances outputted from the actual target domain distribution and the ones generated from the generator network. Thus, this architecture is often referred to in the literature as the “conditional GAN”. One of the most recently successful ADA architectures is the Cycle-Consistent GAN (CycleGAN) [13] architecture. In CycleGAN, it is essentially comprised of two conditional GAN networks. The first network works on the transformation from the source domain ($S$) to the target domain ($T$), $S\rightarrow T$, while the other one works on the transformation in the opposite direction, $T\rightarrow S$. The additional contribution for CycleGAN architecture was the introduction of a new loss function they call it the cycle-consistency loss function. This new loss function assures that if the two conditional GANs networks are connected, they will produce the following identity mapping: $S\rightarrow T\rightarrow S$.

II-B DA Between Synthetic and Real for Perception Tasks

In the context of traffic environments, a number of perceptions tasks has been utilising the DA approach to bridge the gap between real domains from physical sensors and synthetic domains from simulated sensors [13, 24, 25]. It is worth noting that all of these works were only exploring one type of sensors which was cameras either RGB (monocular/stereo) or thermal. For example, in [13], a number of DA between different domains were introduced based on the CycleGAN architecture. For instance, they addressed the semantic segmentation task between the day and night domains on unpaired visual images from multiple road-based datasets. Similarly, in [24], Atapour et al. trained a ConvNet model on synthetic depth and RGB images from the famous game GTA in order to estimate a synthetic monocular depth image. In the testing/inference phase, they took an input real RGB image from the KITTI dataset [26] and with the help of a CycleGAN architecture, they transformed the real RGB image into a synthetic GTA game like RGB image. Then, they passed the synthetic RGB image to their initial trained model to estimate a synthetic depth image. Eventually, they used the same CycleGAN network again to adopt the estimated depth image from the synthetic image domain to a real RGB image domain.

On the other hand, in [25] Zhang et al. proposed deep-learning based approach for thermal infra-red object tracking. To overcome the scarcity of thermal images dataset, they utilised DA based on the CycleGAN architecture to transform images from visual domain to the thermal infra-red domain.

III Proposed Methodology

The main focus of this work is to provide a framework for bridging the gap between real and synthetic point cloud data represented as BEV images for the vehicle detection task. That being said, the same framework can still be used for other perceptions tasks on point cloud data such as semantic segmentation or object tracking. In this section, we will first provide our formulation for the problem at hand. Then subsequently, we will break-down the building blocks of the proposed framework.

III-A Problem Formulation

ConvNet-based architectures for object detection from BEV point cloud data has been achieving state-of-the-art results in many benchmarks [14]. However, with the available insufficient numbers of annotated BEV point cloud data for training such architectures, the trained models are still performing poorly especially in challenging scenarios. The utilisation of annotated synthetic BEV point cloud data from simulated traffic environments could be the key to increase the performance of such models. However, due to the domain shift between real and synthetic BEV point cloud data, the trained model on synthetic data is not necessarily guaranteed to generalise on the real data [10].

Thus, in our formulation for the vehicle detection task from real BEV point cloud data, we are proposing a framework for DA between synthetic BEV point cloud data and real BEV point cloud data. In the first stage of our framework, we train a CycleGAN model between unpaired synthetic BEV point cloud data and real BEV point cloud data. The trained model, in returns, learns a transformation from synthetic BEV point cloud data to real BEV point cloud data and vice versa. As a result, given any annotated synthetic BEV point cloud dataset with vehicles, the trained CycleGAN model will transform that dataset to an annotated real-like BEV point cloud data. Finally, using the transformed dataset, we could train another ConvNet-based model for the vehicle detection task in real BEV point cloud data.

III-B Deep Unsupervised DA via Cycle-Consistent GANs

As we earlier mentioned in Section II-B, the CycleGAN architecture has recently shown promising results in a number of DA tasks between real and synthetic visual domains. Thus, in this work, we will be exploring the CycleGAN architecture for the task of DA between real BEV point cloud domain and synthetic BEV point cloud domain. One of the advantages of the CycleGAN architecture in the context of DA is it can learn transformation between source and target domains without any supervised one-to-one mapping between the two domains. This is beneficial for our task because it is almost impossible for us to have the same traffic scenario and environment captured in both real BEV point cloud data and synthetic BEV point cloud data. However, we can have a handful amount of BEV point cloud data from each domain separately that represent the distribution of that domain.

More formally, given our two domains $S,R$ of the synthetic and the real BEV point cloud data domains. Then, the objective of our adopted CycleGAN-based DA approach (shown in Fig. 4) is to map between the distributions $s\sim\mathbb{P}_{d}(s)$ and $r\sim\mathbb{P}_{d}(r)$ from the synthetic and the real BEV point cloud domains respectively. The proposed CycleGAN-based DA approach achieve this mapping via the two generators, $G_{S\rightarrow R}$ and $G_{R\rightarrow S}$ and the two discriminators $D_{S}$ and $D_{R}$. The generator $G_{S\rightarrow R}$ will try to map the input source synthetic BEV point cloud image to some target real BEV point cloud image. While the generator $G_{R\rightarrow S}$ is trying to map the generated BEV point cloud image from the real target domain back to its original source domain. The discriminator $D_{S}$, on the other hand, is trying to differentiate between a BEV point cloud image $s\in S$ and a generated BEV point cloud image from $G_{R\rightarrow S}$. Conversely, the discriminator $D_{R}$ will be trying to distinguish between a BEV point cloud image $r\in R$ and a generated BEV point cloud image from $G_{S\rightarrow R}$. The two generators networks are deep ConvNet models.

The main building blocks of them are three blocks, namely the encoder, the transformer and the decoder respectively. The encoder’s job is to extract features on multiple levels progressively by down-sampling them from the input BEV point cloud image from both domains. The transformer, on the other hand, takes the extracted features vector encoder in the source domain and transform it into another feature vector in the opposite target domain. The decoder finally up-sample the transformed features vector back to the original shape and dimensionality as it was before going through the encoder. The architecture we used for that combination of encoder, transformer and decoder of our generator networks is based on the architecture proposed in [27]. The encoder in this architecture consists of two convolution layers, while the transformer consists of nine ResNet blocks and the decoder consists of two de-convolution/transposed convolution layers. The two discriminators architecture is a deep ConvNet model as well. They are based on the PatchGAN architecture from [28], which consists of three consecutive convolution layers for feature extraction in patches and a final 1D-convolution layer for the decision whether its input BEV point cloud image is fake or not.

In order to train the proposed CycleGAN-based DA approach for our task, we will be utilising the adversarial loss for the two generators that we have discussed above along with their corresponding discriminators. The first loss for the transformation from domain $S$ to domain $R$ is as follows:

[TABLE]

where $S$ is the synthetic BEV point cloud data domain and $\mathbb{P}_{d}(s)$ is its data distribution.

Similarly, the second loss for the transformation from domain $R$ to domain $S$ is as follows:

[TABLE]

Additionally, in order to penalise the generators of the trained model to generate more realistic BEV point cloud data from each domain $S$ and $R$, the following third loss is added.

[TABLE]

where $\mathcal{L}_{cyc}$ is the cycle-consistency loss which ensures the identity mapping of the each transformed sample BEV point cloud image back to its original source.

Given the three losses from Eq. 1, 2, 3, the objective loss function for the proposed CycleGAN-based DA approach is as follows:

[TABLE]

where $\lambda$ is equal to 10 which was chosen empirically.

Finally, since the objective of training any deep ConvNet model is to minimise a certain loss function, which in our case is the joint loss function in Eq. 4. Thus, we will be using the Adam optimiser for minimising our objective joint loss function using a learning rate of 0.001.

III-C Vehicle Detection in BEV Point Cloud Data via YOLOv3

For the vehicle detection task, we will be the adopting state-of-the-art single stage deep ConvNet architecture for object detection, You Only Look Once (YOLOv3) architecture. Internally, YOLOv3 relies on k-means clustering to have prior bounding boxes “anchors” of a potential region of interests (ROIs) in the input image which goes through a total of 53 convolution layers to extract features from them on 3 different scales. YOLOv3 in returns predicts the four coordinates for the bounding box, an objectness score for each bounding box, and class score for the object that the bounding box may contain. The four coordinates are predicted using a sigmoid function. The objectness score is predicted using a logistic regression which is set to 1 if the bounding box of one of the anchors overlaps with a ground truth bounding box. The class score of a bounding box is predicted via multinomial logistic classifiers which is better than the traditional soft-max classifier when it comes to multi-label classification task such as object detection.

More specifically, in our vehicle detection task from BEV point cloud images, we relied on the YOLOv3-416 derivative architecture, which as the name implies works on input images with a resolution of $416H\times 416W$.

IV Experiments

In this section, we will firstly discuss the datasets we have used for training and validating our trained models. Secondly, the performance of our models will be quantitatively and qualitatively evaluated.

IV-A Datasets

For the task of the DA between synthetic and real BEV point cloud images, we relied on two datasets. The first dataset is the recently released Motion-Distorted LiDAR Simulation (MDLS) dataset introduced in [9]. This dataset represents the synthetic domain $S$ of our CycleGAN-based DA approach discussed in Section III-B. The MLDS dataset was generated from high fidelity simulated urban traffic environments from the CARLA simulator [29] using a simulated Velodyne HDL-64E sensor. The dataset is originally meant for studying the effect of the motion distortion resulted from a moving vehicle-based 3D LIDAR sensor on the generated point cloud data. The dataset consists of two sequences of point cloud data from urban traffic environment involving between 60 to 90 moving vehicle, each one with an average duration of five minutes which results in total 6K point cloud scans. The dataset was annotated with the position of the vehicles in the scene. For our DA task, we first preprocessed the point cloud scans in order to get a BEV image of each scan according to the method introduced in [14]. As a result, we get a total of 6K BEV point cloud images similar to the right image shown in Fig. 1. The second dataset we utilised for the real domain $R$ of our CycleGAN-based DA approach is the BEV benchmark data from the KITTI dataset [11]. The BEV benchmark data consists of 7481 training images and point cloud scans and 7518 test images and point cloud scans. The point cloud data was captured using a real 3D LiDAR sensor the Velodyne HDL-64E sensor. The dataset contains annotations for multiple objects in the traffic scene such as vehicles, pedestrians and cyclists. Similar to the pre-processing step we have done for the MLDS dataset we did it as well for the KITTI dataset in order to get BEV point cloud images like the one shown on the left in Fig. 1. In our experiments for training our CycleGAN-based DA approach, we used a total 6K BEV point cloud images from the MLDS dataset and the 7481 BEV point cloud images of the training split from the KITTI dataset.

Similarly, for the task of the vehicle detection from BEV point cloud images we used the same aforementioned two datasets (MLDS and KITTI) in addition to the domain adapted BEV images from synthetic to real for training our YOLOv3 model. Since our ultimate goal in the vehicle detection task is to identify vehicles in real BEV point cloud images. Thus, we further split the total 7481 real BEV images from the KITTI dataset into 4K for training our YOLOv3 model and 3481 for testing the model.

IV-B Results and Discussion

Firstly, in order to evaluate the effectiveness of our proposed CycleGAN based DA approach for the vehicle detection task from real BEV point cloud images. In fig. 3, we show qualitative results of the trained CycleGAN-based DA approach between synthetic and real BEV point cloud images. In the first row of the figure is the input synthetic BEV point cloud image to our model. The second row represents the output from the generator $G_{S\rightarrow R}$ of our trained CycleGAN model. The third row shows one sample of a real BEV point cloud image from the KITTI dataset. As it can be noticed, the generated BEV point cloud from our CycleGAN model is mimicking and trying to be consistent with the same structure exist in the real BEV point cloud image from KITTI. More specifically, the generated image captures pretty well the structure of the vehicles and the distortion/noise artefacts from resulting from the real Velodyne 3D LiDAR sensor.

For having more quantitative evaluation of our proposed CycleGAN based DA approach for the vehicle detection task, we trained two YOLOv3 models, the first one $YOLO_{S}$ is trained using the 6K synthetic BEV point cloud images, while the other one $YOLO_{R}$ is trained using the same 6K BEV point cloud images but the DA versions of them after feeding them to our trained CycleGAN model and getting its predicted DA real BEV point cloud images. Furthermore, we trained three additional YOLOv3 models with the only difference in the type of training data. The first model $YOLO_{K}$ which as the name implies is trained on the 4K training split BEV point cloud images from the KITTI dataset. The second model $YOLO_{KS}$ is trained using on the 4K images from the KITTI dataset with an additional 6K synthetic BEV point cloud image from the MLDS dataset. The third and final model $YOLO_{KR}$ is trained using the same amount of data to the $YOLO_{KS}$ model, however instead of the MLDS synthetic BEV images we used the DA version predicted from our CycleGAN model.

In Table I, we report the performance of the total 5 YOLOv3 models we mentioned earlier when all are tested on the same 3481 testing real BEV point cloud images from the KITTI dataset. The evaluation metric we used is the average precision score (AP) which summarises the precision-recall curve that commonly used for evaluating object detectors. As it can be noticed from the table, the $YOLO_{R}$ model outperformed the $YOLO_{S}$ with more than 4% in AP score which proves our claim that our CycleGAN-based DA approach for the BEV point cloud images are more efficient than pure synthetic ones for the vehicle detection task. Additionally, the best performing model with 64.29% in AP score is the $YOLO_{KR}$, which again proves the benefits of using domain adapted BEV point cloud images over the purse synthetic ones. This prevalent from Table I by the low AP scores from the $YOLO_{K}$ and the $YOLO_{KS}$ models which achieved only AP score of 57.26% and 59.16% respectively.

For a qualitative measuring of the performance of the trained YOLOv3 models, in Fig. 4, we show a) input sample BEV point cloud image, b), c) and d) the detected bounding boxes (in green colour) from models $YOLO_{K}$, $YOLO_{KS}$ and $YOLO_{KR}$ respectively. The ground truth annotations are highlighted in the light blue colour, while the false or miss-detected objects are highlighted in red colour. As it can be shown, our model $YOLO_{KR}$ gives an accurate detection with the lowest false-positive rate.

V Conclusion

In this work, we have introduced a framework for domain adaptation between synthetic and real BEV point cloud images for the vehicle detection task. The proposed framework utilises deep generative adversarial networks, CycleGAN for the domain adaptation task. Then, given the domain adapted BEV point cloud images we trained a series of object detection models based on state-of-the-art deep ConvNet-based model, YOLOv3. The trained models have shown the effectiveness of the proposed DA approach for the vehicle detection task from real BEV point cloud images. Furthermore, we have evaluated the performance of the trained models on the testing split from real BEV point cloud images from the KITTI dataset. The best performing model was the one utilising our domain-adapted BEV point cloud images which achieved the highest average precision score of 64.29% with an improvement of more than 7% over the compared baseline approaches.

Acknowledgement

This research was fully supported by the Institute for Intelligent Systems Research and Innovation (IISRI) at Deakin University.