Incorporating Human Domain Knowledge in 3D LiDAR-based Semantic Segmentation

TL;DR

This paper introduces a novel approach for 3D LiDAR semantic segmentation that incorporates human domain knowledge into neural networks, reducing manual annotation needs and enhancing training efficiency.

Contribution

It proposes a method to embed human knowledge via pretraining with rule-based classifier samples, improving performance with fewer annotations.

Findings

Pretrained models outperform randomly initialized ones.

Our method achieves comparable accuracy with fewer manual annotations.

Incorporating human knowledge enhances training efficiency.

Abstract

This work studies semantic segmentation using 3D LiDAR data. Popular deep learning methods applied for this task require a large number of manual annotations to train the parameters. We propose a new method that makes full use of the advantages of traditional methods and deep learning methods via incorporating human domain knowledge into the neural network model to reduce the demand for large numbers of manual annotations and improve the training efficiency. We first pretrain a model with autogenerated samples from a rule-based classifier so that human knowledge can be propagated into the network. Based on the pretrained model, only a small set of annotations is required for further fine-tuning. Quantitative experiments show that the pretrained model achieves better performance than random initialization in almost all cases; furthermore, our method can achieve similar performance with fewer manual annotations.

Tables

Table 1. TABLE I: The samples of manual annotations.

eople	car	trunk	bush	building	cyclist	unknown
training set	1533	6014	1837	9064	7736	366	5113
testing set	1880	5074	1746	9102	3230	562	4630

Table 2. TABLE II: The performance of rule-based classifier.

eople	car	trunk	building	unknown
labeling on training set	3009	6255	5914	3596	9723
F1 score on testing set	50.0	64.8	48.9	62.6	66.0

Table 3. TABLE III: The comparisons with F1 measure on testing set.

classifier	people	car	trunk	building	unknown	mean score
rule-based	50.0	64.8	48.9	62.6	66.0	58.5
pretrained-CNN	57.4	67.0	50.4	66.0	67.2	61.6

Table 4. TABLE IV: The subdivision of training set for funetuning.

finetuning data	people	car	trunk	bush	building	cyclist	unknown
training set	1533	6014	1837	9064	7736	366	5113
sub-100	100	100	100	100	100	100	100
sub-400	400	400	400	400	400	366	400
sub-1600	1533	1600	1600	1600	1600	366	1600

Table 5. TABLE V: The F1 score of different classifiers on testing set.

finetuning data	classifier	people	car	trunk	bush	building	cyclist	unknown	mean score
sub-100	baseline-100	46.0	56.5	66.3	65.5	56.0	30.8	35.8	51.0
	pretrain-100	55.0	71.6	68.6	63.9	61.8	31.3	39.4	55.9
sub-400	baseline-400	59.4	72.2	73.5	71.4	71.8	44.0	35.1	61.1
	pretrain-400	67.1	79.7	71.9	72.8	70.8	45.5	48.5	65.2
sub-1600	baseline-1600	68.7	80.3	74.4	70.8	69.9	46.1	48.9	65.6
	pretrain-1600	71.2	82.5	75.9	75.9	72.6	46.5	49.0	67.7
training set	baseline-all	69.2	85.1	77.2	75.3	76.8	44.6	53.8	68.8
	pretrain-all	71.6	87.5	80.2	77.1	78.2	45.4	53.6	70.5

Equations

$$f_{\theta }:x\to y\in \{1,...,K\}$$

$$\begin{array}{rl}& {\theta }^{\ast }={\mathrm{argmin}}_{\theta }L(\theta ;X,Y)\\ & L(\theta ;X,Y)=-\frac{1}{M}\sum _{i=1}^M\sum _{k=1}^K\text{1}[y_i^k=1]ln(P_{\theta }^k(x_i)),\end{array}$$

$${\theta }^{\ast }={\mathrm{argmin}}_{\theta }L(\theta ;X,Y,{\theta }_0),$$

$${\theta }_r={\mathrm{argmin}}_{\theta }L(\theta ;X_r,Y_r,{\hat{Y}}_r,{\theta }_0),$$

$${\theta }^{\ast }={\mathrm{argmin}}_{\theta }L(\theta ;X_l,Y_l,{\hat{Y}}_l,{\theta }_r),$$

$$F1-Measure=\frac{2\ast recall\ast precision}{recall+precision}\cdot 100.$$

Full text

I Introduction

Recently, 3D LiDAR sensors have been widely implemented as the “eyes” of autonomous driving systems[1]. How to achieve efficient scene parsing, e.g., semantic segmentation, based on 3D LIDAR data has attracted increasing attention. Generally, semantic segmentation is the procedure of finding an object label for each point or data cluster[2]. In this work, the problem is defined as classification based on segmentation, as shown in Fig. 1. Raw 3D LiDAR data can be equally represented by a range image (b) in the polar coordinate system, where the pixel value is the range distance. Then, oversegmentation is conducted on the range frame (c), and the problem of semantic segmentation is solved by discriminating the label of each segment (d).

3D LiDAR-based semantic segmentation has been studied for the past decade[2, 3, 4]. The traditional methods use handcrafted features[3] that have a clear definition in the real world, e.g., the width is used to distinguish people from cars. Thus, these methods are interpretable. However, the adaptability of features for different scenes remains a challenge, and expert knowledge is required to adjust the parameters of the classifier.

The outstanding performance of deep learning methods in image semantic segmentation[5] has encouraged researchers to apply these methods to 3D LiDAR data. These data-driven methods avoid handcrafted features by using abundant annotated data. However, the generation of fine annotations, especially for 3D LiDAR data, is challenging, and few public datasets are available for 3D LiDAR-based semantic segmentation aimed at autonomous driving applications.

The two aforementioned types of methods have their own advantages and disadvantages. Traditional methods rely on human domain knowledge, while deep learning methods are data driven. Is there a way to combine the advantages and make up for the shortcomings? To the best of the author’s knowledge, two techniques have been reported in the literature. The first is a semisupervised approach[6] that converts human prior knowledge into constraint information, such as data associations between frames, and adds the constraint item to the loss function. We have verified the effectiveness of this method in our previous work[7]. The second is pretraining, e.g., initializing new networks with parameters trained on IMAGENET[8], which is widely used in tasks related to image processing. The effectiveness of pretraining is discussed in detail in [9]. However, it is unreasonable to initialize the network parameters directly from an image processing task, which motivates us to design a pretraining method suitable for 3D LiDAR data.

In this paper, we make full use of the advantages of traditional methods and neural network methods via incorporating human domain knowledge into the neural network model to reduce the demand for large numbers of manual annotations and improve the training efficiency. The proposed method consists of two steps: parameter pretraining and parameter fine-tuning. In first step, a rule-based classifier based on human knowledge is designed, and samples are fed through the classifier to perform unsupervised classification. Then, these auto-annotated data are used to pretrain a convolutional neural network (CNN) to ensure that the parameters of the CNN fit the human knowledge. In the second step, the parameters in the pretrained CNN are transferred to a new network. Because the new CNN is well initialized, only a small number of manual annotations is necessary to update the parameters. We evaluate this method on a dynamic campus scene. Quantitative experiments show that the pretrained method has better performance than random initialization in almost all cases; furthermore, our method achieves similar performance with fewer manual annotations.

The remainder of this paper is structured as follows. Sect. II discusses related work. The proposed method is presented in Sect. III. Sect. IV shows the implementation details. Sect. V presents the experimental results. Finally, we draw conclusions in Sect. VI.

II Related Works

Semantic segmentation for 3D LiDAR data is not a new topic. We firstly review the methods on semantic segmentation, then discuss how to incorporate human knowledge.

II-A Semantic Segmentation

A popular method of semantic segmentation is classification of each point or data cluster. We separate the related literature into traditional methods and deep learning methods.

In traditional methods, some researchers assume that each point or data cluster is independent; for example, [10] presents a versatile framework including feature selection, feature extraction and classification. One-frame LiDAR data usually have millions of points, so the direct method is time consuming. Classification based on a segmentation framework is proposed in [3], where only the label of a cluster is evaluated. Some works consider the spatial relationship between elements, e.g., via the Markov random Field (MRF) or conditional random field (CRF). Features are embedded in node potentials, and spatial relationships are encoded in edge potentials [2]. The solution of a CRF or MRF sometimes requires high-dimensional optimization. Thus, [11] proposes a simplified MRF, where the node and edge potentials are directly updated, and [12] attempts to simplify the point clouds via voxel-neighbor structure. The main disadvantage of traditional methods is the adaptability of handcrafted features to different scenes.

In deep learning methods, features are automatically learned from data. Researchers focus mainly on discussing data representations and new network structures. Inspired by image semantic segmentation, raw 3D data can be converted into 2D images. [13] uses virtual 2D RGB images obtaine via Katz projection, and [4, 14] unwrap 3D LiDAR data on spherical range image. Another stream of research considers 3D representations. Voxel occupancy grid is a good way to make irregular raw LiDAR data grid-aligned [15]. The voxel representation is further improved in OctNet[16], which has more efficient memory allocation and computation. New network structures specified for 3D data are also studied. PointNet[17] directly takes raw point clouds as input, and a novel type of neural network is designed with multilayer perceptrons (MLPs). Based on [17], [18] extends the method to incorporate larger-scale spatial context, and [19] proposes the superpoint graph to capture the contextual relationships from point clouds. The main shortcoming of deep learning methods is the demand for large amounts of manual annotations.

II-B Incorporating Human Knowledge

The purpose of incorporating human knowledge is to reduce the need for large numbers of manual annotations in deep learning methods. To the best of the author’s knowledge, two types of methods have been reported in the literature: semi/weakly-supervised learning and model pretraining.

Semi/weakly-supervised learning involves introducing a few fine annotations or large numbers of ambiguous annotations during parameter learning. [20] uses point supervision, where annotators are asked to point to an object if one exists. Then, the point annotation and objectness prior are incorporated into the loss function to train the neural network. Similar work is proposed in ScribbleSup[21], which trains convolutional networks for semantic segmentation supervised by scribbles. The size constraint of an object is considered in [22]. Our previous work on semisupervised learning implements a pairwise constraint, which encourages associated samples to be assigned the same labels[7]. Some works focus on training models in weakly supervised settings, such as image-level tags[23], bounding box labels[24] and unlabeled examples[25]; these weak supervision methods can be applied separately or in combination.

During model pretraining, the target network is initialized with the parameters from a pretrained baseline network, e.g., the parameters in the first n layers copy from the baseline network and the remaining parameters are set randomly. This strategy has been widely used in image processing tasks, e.g., segmentation[26] and transfer learning[27]. The baseline network is usually trained with IMAGENET[8]. The experiments in [9] confirm and clarify the advantages of unsupervised pretraining, which provides a good initial marginal distribution and exhibits properties of a regularizer. [28] introduces human priors by pretraining a model to regress a cost function under the framework of inverse reinforcement learning. Inspired by [28], we propose a pretraining method suitable for 3D LiDAR data in which large numbers of auto-annotated data are generated with human domain knowledge.

III METHODOLOGY

III-A Problem Definition

Let $P$ be the range frame converted from raw 3D point clouds. Segments $\{s_{i}\}_{i=1}^{N}$ are obtained on $P$ by evaluating the similarity of 3D points with their neighborhoods, e.g., a region growing method. We assume that one segment $s$ measures only a single object after oversegmentation. As $s$ commonly represents a part of the object, a data sample $x$, including $s$ and the surrounding data, is defined at the center of $s$, as shown in Fig.2. In one range frame $P$, $\{s_{i}\}_{i=1}^{N}$ and $\{x_{i}\}_{i=1}^{N}$ can be equally converted into each other. The problem in this work is formulated as learning a multiclass classifier $f_{\theta}$ that maps $x$ to a label $y\in\{1,...,K\}$ and subsequently associates $y$ with the 3D points of $s$.

[TABLE]

Given a set of annotated data $X=\{x_{i}\}_{i=1}^{M},Y=\{y_{i}\}_{i=1}^{M}$, where $\{y_{i}\}$ is a one-hot label for $\{x_{i}\}$, a common way of learning a classifier $f_{\theta}$ is to find the best $\theta^{*}$ that minimizes a loss function $L$, i.e., the cross entropy, as below.

[TABLE]

where $\textbf{1}[*]$ is an indicator function and $P^{k}_{\theta}(x_{i})$ is the probability that $x_{i}$ is assigned a label $k$ by a classifier with parameters $\theta$. Stochastic gradient descent (SGD) is commonly applied to solve $\theta^{*}$. Thus, Equation (2) can be rewritten as:

[TABLE]

where $\theta_{0}$ is the start position of SGD. $\theta_{0}$ can be obtained from a random distribution, e.g., a truncated normal distribution; or initialized from human knowledge via pretraining, such as in the proposed method.

III-B Work Flow

As shown in Fig. 2, the framework consists of two steps: parameter pretraining and parameter fine-tuning. One range frame is divided into multiple segments, and each segment corresponds to one sample $x$ in Fig. 3. Consequently, we can automatically produce a large number of samples. The sample generation follows that of [7].

During the pretraining step, we design a rule-based classifier that incorporates human knowledge. The unlabeled samples $X_{r}$ are passed through this classifier to predict the label; thus, the auto-annotated data $(X_{r},Y_{r})$ are obtained. The rule-based classifier works in an unsupervised manner, and we need not train the classifier via manual annotations. Combining $X_{r}$ and $Y_{r}$, a CNN is pretrained with a random initialization $\theta_{0}$, and the parameters are updated via back-propagation.

[TABLE]

where $\widehat{Y}_{r}$ is the output of the CNN. In this way, we obtain the pretrained parameter $\theta_{r}$. $\widehat{Y}_{r}$ will continue to fit $Y_{r}$, so we assume that the human rules can be propagated into the CNN.

In the fine-tuning step, the pretrained parameters $\theta_{r}$ are first transferred into a new neural network. Since the new classifier is well initialized, we need only a small number of manual annotations $(X_{l},Y_{l})$ for parameter fine-tuning. The parameter updating is the same as in the first step, except that the start position of the optimization is different. The final classifier is obtained by:

[TABLE]

where $\widehat{Y}_{l}$ is the output of the CNN and $\theta_{r}$ is the start position.

III-C Rule-based Classifier

LiDAR can directly measure distance information without being affected by illumination, and this robust attribute motivates us design a classifier based on rules in the real world. The sample and segment are associated as detailed in III-A; thus, only features of the segment are considered for each sample in the rule-based classifier. As shown in Fig. 4, the height and width of a segment are calculated from the raw point cloud, and these two features reflect the physical attributes of objects in the real world, i.g., the height of a car generally does not exceed 2 m. We believe that the width and height of objects are efficient information to design a simple classifier. As shown in Algorithm 1, a trunk is higher than 2 m and has a width in (0, 2.5], people are shorter than 2 m but have widths larger than 0.2 m, a car is shorter than 2 m and has a width in [1.5,2.5], and so on. These rules can easily be understood by a human, however, if we want the CNN to understand these rules, the general approach is to train the CNN with a large number of manual annotations. The proposed rule-based classifier can automatically generate annotations without human effort, which accelerates the CNN learning.

III-D Parameter Pretraining

Parameter pretraining has been widely used in image processing tasks such as classification, detection and segmentation, but it is unreasonable to initialize a network toward LiDAR data using the parameters from image processing tasks, as these two types of data are substantially different in terms of both human visual and physical meaning. Therefore, a pretraining approach should be designed for LiDAR data. Now, the question is why pretraining reduces the need for manual annotations during the training phase of neural networks? We answer the question from two perspectives.

Perspective of probability. Human knowledge describes the distribution of the input data $X$, i.e., $P(X)$, which represents data priors, and the target CNN classifier can be treated as a conditional probability that predicts the label $Y$ for each input data, i.e., $P(Y|X)$. In general, the CNN is trained with only human annotations that are similar to the joint distribution $P(X,Y)$, but data priors $P(X)$ are ignored. Based on Bayes rule, $P(Y|X)=P(Y,X)/P(X)$, if accurate priors are supported, we believe that the dependence on manual annotation can be reduced. We design a rule-based classifier to obtain $P(X)$ from human domain knowledge.

Perspective of gradient descent. Gradient descent is the conventional parameter updating method of CNNs. The selection of the initial position largely determines whether gradient descent can converge to the global minimum. Random initialization is a common strategy. The initial position is randomly selected in the high-dimensional parameter space, which increases the possibility of training results falling into local minima. In our method, the samples used for pretraining are supervised by human rules, that is, autogenerated from the rule-based classifier. Thus, the network will continue to fit these rules, and the performance of the pretraining network is related to the rule-based classifier. Although we cannot assume that the pretrained parameters are optimal, they are reasonable. When the initial position of gradient descent starts from the pretrained parameters, the dependence on manual annotation is reduced.

In our method, the samples for pretraining are autogenerated by the rule-based classifier, and we do not require any human annotation in this step.

III-E Parameter Fine-tuning

We define parameter fine-tuning as initializing a new network with pretrained parameters and training the network with manual annotations. The main body of a CNN consists of convolutional layers and fully connected (FC) layers. Generally, there are two ways to perform parameter fine-tuning in the context of a CNN. Both of them copy the parameters of convolutional layers from a pretrained network to the new network and randomly set the FC layers. The difference is whether the paramters in th convolutional layers are fixed during fine-tuning. We test these two configurations in experiments.

IV Implementation Details

The CNN used here consists of three convolutional layers whose dimensions in [width,height,depth] are [256,256,32], [128,128,32], and [64,64,64]; two fully connected layers whose dimensions are both [128,1]; and one softmax layer. In parameter fine-tuning, the CNN predicts 7 labels: people, car, trunk, bush, building, cyclist and unknown. In pretraining, the CNN predicts only 5 labels, namely, people, car, trunk, building and unknown since the rule-based classifier cannot fully discriminates cyclist and bush.

All networks are trained under the TensorFlow framework using ADAM solver and a learning rate of 1e-4. The batch size is 2, and we save a checkpoint every 100 iterations. The training phase stops when the loss converges or is less than 1e-4. For each classifier, we evaluate all checkpoints on the testing set and select the checkpoint with the highest F1 score.

V EXPERIMENTAL RESULTS

V-A Data Set

The performance of the proposed method is evaluated on a dynamic campus dataset collected by an instrumented vehicle with a GPS/IMU suite and a Velodyne-HDL32, as shown in Fig. 7. The total route is approximately 890 meters. All sensor data are collected, and each data frame is associated with a time log for synchronization. The GPS/IMU data are logged at 100 Hz. The LiDAR data are recorded at 10 Hz and include 1039 frames of training data (red line in Fig. 5) and 790 frames of testing data (black line in Fig. 5). One frame can produce multiple samples; for example, we obtain 6014 car samples from the 1039 frames of training data in TABLE I.

To make quantitative comparisons, manual annotations[7] are conducted on both training and testing sets; the labeling results are shown in TABLE I.

V-B Rule-based Classifier

The rule-based classifier is simple, and its effectiveness should be assessed before conducting further experiments. As shown in TABLE. II, we pass both training and testing sets through this classifier which assigns one label for each sample. Currently, the classifier supports only 5 categories. In this way, we collect the auto-annotated training set (the second row of TABLE. II) for parameter pretraining. At the same time, the effectiveness is evaluated on the testing set in the term of F1 measure, which is defined as:

[TABLE]

We merge bush and cyclist in TABLE. I into unknown when calculating the F1 score for the rule-based classifier, and the results are shown in the third row of TABLE. II.

The F1 scores in TABLE. II are encouraging: simple rules ensure reasonable results, and the rule-based classifier is effective. The confusion matrix is shown in Fig. 6 (a), one notable result is that the recall of trunk is very high.

V-C Pretraining Results

The CNN is trained with the autolabeled samples in TABLE. II. We expect the pretrained CNN to have similar performance to the rule-based classifier since the samples are supervised by the rules. As illustrated in TABLE. III, the pretrained CNN has better F1 scores on each category than does the rule-based method. A more detailed comparison is in Fig. 6. These two classifier has very similar performances, i.g., they both achieve high recalls on trunk. On the basis of these results, we assume that the rules designed by humans have propagated into the pretrained CNN and that the parameters of CNN are more reasonable than random initialization.

We emphasize that pretraining dose not require any manual labeling. Although the samples are supervised by rules, the pretrained CNN still performs better than the rule-based classifier.

V-D Fine-tuning Results

The fine-tuning data has four components as shown in TABLE. IV. Sub-100$\sim$1600 are randomly selected from the raw training set. For example, sub-1600 means choosing 1600 samples from each category. If the number of sample is less than 1600, such as for people and cyclist categories, we do not perform any sample augmentation.

As illustrated in TABLE. V and Fig. 7, different classifiers are tested based on the fine-tuning data. Baseline-* means the parameters are initialized with the truncated normal distribution, and pretrain-* means the parameters in the convolutional layers copy from the pretrained network. All classifiers share the same network structure detailed in Sect. IV.

First, we discuss the results of a few annotations. our method performs better than the random version under sub-100 and sub-400; furthermore, the mean score of pretrain-400 is near that of baseline-1600, which illustrates the potential of our method to achieve high performance fewer manual annotations. Second, from the perspective of category scores, the highest scores belong mostly to the classifiers initialized by human rules. Third, from the perspective of mean scores as shown in Fig. 8, the pretrained versions have higher scores than the random versions under the same manual annotations: pretrain-400 is near baseline-1600, and pretrain-1600 is near baseline-all. Another notable result is that the gap between random and pretrained initialization becomes small as the number of manual annotations increases. In conclusion, human rules help to reduce the demand for manual annotations.

Two methods of parameter fine-tuning are discussed in Sect. III-E. Pretrain-* in Fig. 8 indicates the first way, where the parameters of convolutional layers are updated during fine-tuning, and the pretrain-fix-* indicates that the convolutional parameters are initialized from the pretrained network and are fixed during fine-tuning. We find that the F1 scores of pretrain-fix-100 are higher than those of pretrain-100, but as the manual annotations increases, the performance of pretrain-fix trends to become stable, e.g., pretrain-fix-1600 and pretrain-fix-all have almost the same score. These results shows that the fixed fine-tuning method has better adaptability for cases with very few annotations.

VI conclusion and future work

In this paper, we propose a new method aimed at semantic segmentation based on 3D LiDAR data. To reduce the substantial demand for manual annotations during parameter training, we attempt to incorporate human knowledge into a neural network via parameter pretraining. To this end, we first pretrain a model with the autogenerated samples from a rule-based classifier so that human knowledge can be propagated into the network. Based on the pretrained model, only a small set of annotations are required to perform further finetuning. This method is examined extensively on a dynamic scene. The promising results indicate reduced reliance of manual annotation. Future work will consider the addition of more priors/knowledge, e.g., the spatial and temporal relationships between samples.