Transferable Semi-supervised 3D Object Detection from RGB-D Data

TL;DR

This paper introduces a semi-supervised 3D object detection method that transfers knowledge from classes with 3D labels to those with only 2D labels, improving detection accuracy on RGB-D datasets.

Contribution

The authors propose a novel transferable semi-supervised model with specialized loss functions and a Box-to-Point Cloud Fit network for effective knowledge transfer.

Findings

Outperforms baseline methods on SUN-RGBD and KITTI datasets.

Enables 3D detection for classes with only 2D annotations.

Enhances fully-supervised methods with the Box-to-Point Cloud Fit network.

Abstract

We investigate the direction of training a 3D object detector for new object classes from only 2D bounding box labels of these new classes, while simultaneously transferring information from 3D bounding box labels of the existing classes. To this end, we propose a transferable semi-supervised 3D object detection model that learns a 3D object detector network from training data with two disjoint sets of object classes - a set of strong classes with both 2D and 3D box labels, and another set of weak classes with only 2D box labels. In particular, we suggest a relaxed reprojection loss, box prior loss and a Box-to-Point Cloud Fit network that allow us to effectively transfer useful 3D information from the strong classes to the weak classes during training, and consequently, enable the network to detect 3D objects in the weak classes during inference. Experimental results show that our proposed algorithm outperforms baseline approaches and achieves promising results compared to fully-supervised approaches on the SUN-RGBD and KITTI datasets. Furthermore, we show that our Box-to-Point Cloud Fit network improves performances of the fully-supervised approaches on both datasets.

Figures

Tables

Table 1. Table 1: 3D object detection AP on SUN-RGBD val set. Fully-supervised methods are trained on 2D and 3D box labels of all classes while Cross-category (CC) Semi-supervised methods are trained on 3D box labels of classes in C 3 ​ D subscript 𝐶 3 𝐷 C_{3D} and 2D box labels of all classes. BoxPC, R, P refer to the BoxPC Fit network, relaxed reprojection loss and box prior loss respectively. * refers to our implementation.

	bathtub	bed	toilet	chair	desk	dresser	nstand	sofa	table	bkshelf	mAP
Fully-Supervised:
DSS [27]	44.2	78.8	78.9	61.2	20.5	6.4	15.4	53.5	50.3	11.9	42.1
COG [24]	58.3	63.7	70.1	62.2	45.2	15.5	27.4	51.0	51.3	31.8	47.6
2D-driven [14]	43.5	64.5	80.4	48.3	27.9	25.9	41.9	50.4	37.0	31.4	45.1
LSS [25]	76.2	73.2	73.7	60.5	34.5	13.5	30.4	60.4	55.4	32.9	51.0
FPN [21]	43.3	81.1	90.9	64.2	24.7	32.0	58.1	61.1	51.1	33.3	54.0
FPN*	46.0	80.3	82.8	58.5	24.2	36.9	47.6	54.9	42.0	41.6	51.5
FPN* + BoxPC Refine	51.5	81.0	85.0	59.0	24.5	38.6	52.2	55.3	43.8	40.9	53.2
CC Semi-Supervised:
FPN*	6.1	69.5	28.1	12.4	18.1	17.5	37.4	25.8	23.5	6.9	24.5
FPN* w/o OneHot	24.4	69.5	30.5	15.9	22.6	19.4	39.0	37.3	29.0	12.8	30.1
Ours + R	29.5	60.9	65.3	36.0	20.2	27.3	50.9	46.4	28.4	6.7	37.2
Ours + BoxPC	28.4	67.9	73.3	32.3	23.3	31.0	50.9	48.9	33.7	16.4	40.6
Ours + BoxPC + R	28.4	68.1	77.9	32.9	23.3	30.6	51.1	49.9	34.7	13.7	41.1
Ours + BoxPC + R + P	30.2	70.7	76.3	33.6	24.0	32.5	52.0	49.8	34.2	14.8	41.8

Table 2. Table 2: 3D object detection AP on KITTI val set, evaluated at 3D IoU threshold of 0.25 for all classes.

Method	Cars			Pedestrians			Cyclists
	Easy	Moderate	Hard	Easy	Moderate	Hard	Easy	Moderate	Hard
Fully-Supervised:
FPN [21]	96.69	96.29	88.77	85.40	79.20	71.61	86.59	67.60	63.95
CC Semi-Supervised:
FPN	0.00	0.00	0.00	4.31	4.37	3.74	1.05	0.71	0.71
FPN w/o OneHot	0.08	0.07	0.07	55.66	47.33	41.10	54.35	36.46	34.85
Ours + BoxPC + R	14.71	12.19	11.21	75.13	62.46	57.18	61.25	41.96	40.07
Ours + BoxPC + R + P	69.78	58.66	51.40	76.85	66.92	58.71	63.29	47.54	44.66

Table 3. Table 4: 3D object detection AP on SUN-RGBD val set with varying B u ​ p ​ p ​ e ​ r ∗ subscript superscript 𝐵 𝑢 𝑝 𝑝 𝑒 𝑟 B^{*}_{upper} for the relaxed reprojection loss function.

$B_{lower}^{\ast }$	$B_{upper}^{\ast }$	bathtub	bed	toilet	chair	desk	dresser	nstand	sofa	table	bkshelf	mAP
$B_{2D}^{\ast }$	$1.0B_{2D}^{\ast }$	17.0	11.9	60.1	24.3	11.2	24.0	46.8	30.3	19.2	11.3	25.6
$B_{2D}^{\ast }$	$1.5B_{2D}^{\ast }$	29.5	60.9	65.3	36.0	20.2	27.3	50.9	46.4	28.4	6.7	37.2
$B_{2D}^{\ast }$	$2.0B_{2D}^{\ast }$	16.9	23.4	55.8	35.0	19.6	25.1	49.2	48.3	31.3	3.5	30.8

Equations

$$L_{seg}^{3D}$$

$$L_{box}^{3D}$$

$$+L_{s-cls}^c+L_{s-reg}^c+L_{corner}^c$$

$$\hat{p}=f_{boxpc-cls}(X,B)\in [0,1],$$

$$\Delta \hat{B}=f_{boxpc-reg}(X,B)\in {\mathbb{R}}^7,$$

$$L_{fit}^{2D}$$

$$L_{boxpc}^{3D}=\sum _{c\in C_{3D}}{L}_{cls}^c+L_{reg}^c,$$

$$L_{cls}=\{\begin{array}{ll}H(1,f_{boxpc-cls}(X,B^{\ast }-\delta )),& \text{if δ∈P+},\\ H(0,f_{boxpc-cls}(X,B^{\ast }-\delta )),& \text{if δ∈P−},\end{array}$$

$$L_{reg}=l(\delta ,f_{boxpc-reg}(X,B^{\ast }-\delta )),$$

$$L_{reproj}^{2D}$$

$$\tilde{l}(y_{lower}^{\ast }$$

$$L_{prior}^{2D}=\sum _{c\in C_{2D}}{L}_{vol}^c+L_{s-var}^c,\text{vspace}-0.1in$$

$$L_{vol}^c=\max (0,V^c-{\hat{b}}_{0,h}^c{\hat{b}}_{0,w}^c{\hat{b}}_{0,l}^c),$$

$$L_{s-var}^c=l({\bar{b}}_{0,h}^c,{\hat{b}}_{0,h}^c)+l({\bar{b}}_{0,w}^c,{\hat{b}}_{0,w}^c)+l({\bar{b}}_{0,l}^c,{\hat{b}}_{0,l}^c),$$

$$L_{box}^{2D}=L_{fit}^{2D}+L_{reproj}^{2D}+L_{prior}^{2D},$$

$$L_{seg}$$

$$L_{box}$$

Full text

1 Introduction

3D object detection refers to the problem of classification and estimation of a tight oriented 3D bounding box for each object in a 3D scene represented by a point cloud. It plays an essential role in many real-world applications such as home robotics, augmented reality and autonomous driving, where machines must actively interact and engage with its surroundings. In the recent years, significant advances in 3D object detection have been achieved with fully supervised deep learning-based approaches [32, 6, 5, 7, 17, 41, 36, 13, 21]. However, these strongly supervised approaches rely on 3D ground truth datasets [26, 8, 33, 34] that are tedious and time consuming to label even with good customised 3D annotation tools such as those in [26, 8]. In contrast, calculations based on [18] show that it can be $\sim$3-16 times faster (depending on the annotation tool used) to label 2D than 3D bounding boxes (details in supplementary). This observation inspires us to investigate the direction of training a 3D object detector for new object classes from only 2D bounding box labels of these new classes, while simultaneously transferring information from 3D bounding box labels of the existing classes. We hope that this direction of research leads to cost reductions, and improves the efficiency and practicality of learning 3D object detectors in new applications with new classes.

More specifically, the objective of this paper is to train a 3D object detector network with data from two disjoint sets of object classes - a set of classes with both 2D and 3D box labels (strong classes), and another set of classes with only 2D box labels (weak classes). In other words, our goal is to transfer useful 3D information from the strong classes to the weak classes (the new classes of a new 3D object detection application correspond to these weak classes) . We refer to this as cross-category semi-supervised 3D object detection as illustrated in Fig. 1.

To this end, we propose a novel transferable semi-supervised 3D object detection network. Our network leverages on the state-of-the-art Frustum PointNets [21] as the backbone. We train the backbone network to make class-agnostic segmentations and class-conditioned initial 3D box predictions on the strong classes in a fully-supervised manner. We also supervise the network with 2D box labels of the weak classes using a relaxed reprojection loss which corrects 3D box predictions that violate boundaries specified by the 2D box labels and through the prior knowledge of the object sizes. To transfer knowledge from 3D box labels from the strong classes, we first train a Box-to-Point Cloud (BoxPC) Fit network on 3D box and point cloud pairs to reason about 3D boxes and the fit with their corresponding point clouds. More specifically, we proposed an effective and differentiable method to encode a 3D box and point cloud pair. Finally, the 3D box predictions on the weak classes are supervised and refined by the BoxPC Fit network.

The contributions of our paper are as follows:

• •

We propose a network to perform 3D object detection on weak classes, where only 2D box labels are available. This is achieved using relaxed reprojection and box prior losses on the weak classes, and transferred knowledge from the strong classes.

• •

A differentiable BoxPC Fit network is designed to effectively combine BoxPC representation. It is able to supervise and improve the 3D object detector on the weak classes after training on the strong classes.

• •

Our transferable semi-supervised 3D object detection model outperforms baseline approaches and achieves promising results compared to fully-supervised methods on both SUN-RGBD and KITTI datasets. Additionally, we show that our BoxPC Fit network can be used to improve the performance of fully-supervised 3D object detectors.

2 Related Work

3D Object Detection

3D object detection approaches have advanced significantly in the recent years [25, 24, 14, 32, 6, 5, 7, 17, 41, 36, 13, 21]. However, most approaches are fully-supervised and highly-dependent on 3D box labels that are diffcult to obtain. To the best of our knowledge, there are no existing weakly- or semi-supervised 3D object detection approaches. Recently, [29] proposed a self-supervised Augmented Autoencoder that trains on CAD models, removing the need for pose-annotated training data. However, it is unclear if the method generalizes to general 3D object detection datasets with high intra-class shape variations and noisy depth data.

Weakly- and Semi-Supervised Learning

There is growing interest in weakly- and semi-supervised learning in many problem areas [39, 40, 1, 3, 2, 38, 30, 28] because it is tedious and labor intensive to label large amounts of data for fully supervised deep learning. There is a wide literature of weakly-supervised [12, 20, 31, 42] and semi-supervised [19, 37, 9, 16] learning approaches for semantic segmentation. Both strong and weak labels on the inference classes are required in conventional semi-supervised learning. Hence, the approach remains expensive for applications with new classes. [35, 10, 11] proposed the cross-category semi-supervised semantic segmentation which is a more general approach of semi-supervised segmentation, where the model is able to learn from strong labels provided for classes outside of the inference classes. They outperformed weakly- and semi-supervised methods, and showed on-par performances with fully-supervised methods. Inspired by the effectiveness of the transferred knowledge, we propose to tackle the same problem in the 3D object detection domain.

3 Problem Formulation

Let $C=C_{2D}\cup C_{3D}$, where $C_{2D}$ is the set of classes with only 2D box labels (weak classes) and $C_{3D}$ is the disjoint set of classes with 2D and 3D box labels (strong classes). We tackle the cross-category semi-supervised 3D object detection problem where 3D object detection is performed on the object classes in $C_{2D}$ while training is done on strong labels from $C_{3D}$ and weak labels from $C_{2D}$. In 3D object detection with RGB-D data, our goal is to classify and predict amodal 3D bounding boxes for objects in the 3D space. Depth data can be provided by various depth sensors e.g. RGB-D sensors and LiDAR etc. Each 3D bounding box is parameterized by its center $(b_{x},b_{y},b_{z})$, size $(b_{h},b_{w},b_{l})$ and orientation $b_{\theta}$ along the vertical axis.

4 Method

4.1 Frustum PointNets

In this section, we briefly discuss Frustum PointNets (FPN), the 3D object detection framework which we adopt as the backbone of our framework. Refer to [21] and our supplementary for more details.

4.1.1 Frustum Proposal

As seen in Fig. 2, the inputs to the network is an image with a 2D box for each object and a 3D point cloud. During inference, we obtain the 2D boxes from a 2D object detector trained on $C$. We project the point cloud onto the image plane using the camera projection matrix and select only the points that lie in the 2D box for each object. This reduces the search space to only points within a 3D frustum which we refer to as frustum point cloud. Next, variance of the points in each frustum point cloud is reduced by rotating the frustum about the vertical axis of the camera to face the front. Formally, let us denote a frustum point cloud by $X=[x_{1},x_{2},...,x_{N}]^{T}\in\mathbb{R}^{N\times(3+k)}$, where $N$ is the number of points. $x_{n}\in\mathbb{R}^{3+k}$ (i.e. x, y, z, RGB or reflectance) is the $n$-th point in the point cloud.

4.1.2 3D Instance Segmentation

The frustum point cloud contains foreground points of an object which we are interested in and background points from the surroundings. To isolate the foreground object and simplify the subsequent 3D box estimation task, the frustum point cloud is fed into a 3D instance segmentation network $f_{seg}$ which predicts a class-agnostic foreground probability for each point. These probabilities are used to mask and retrieve the points with high values, thus giving a masked point cloud. $f_{seg}$ is trained by minimizing

[TABLE]

on $C_{3D}$, where $H$ is the point-wise binary cross-entropy loss and $f^{*}_{seg}$ is the ground truth mask.

4.1.3 3D Box Estimation

The 3D box estimation network $f_{box}$ predicts an initial 3D box of the object $\hat{B}_{0}$ given the masked point cloud and a one hot class vector from the 2D detector. Specifically, we compute the centroid of the masked point cloud and use it to translate the masked point cloud to the origin position to reduce translational variance. Next, the translated point cloud is given to two PointNet-based [22] networks that will predict the initial 3D box $\hat{B}_{0}$. The center $(b_{x},b_{y},b_{z})$ is directly regressed by the network while an anchor-based approach is used to predict the size $(b_{h},b_{w},b_{l})$ and rotation $b_{\theta}$. There are $NS$ size and $NH$ rotation anchors, and $3{NS}$ size and $NH$ rotation residuals. Hence, the network has $3+4NS+2NH$ outputs that are trained by minimizing

[TABLE]

on $C_{3D}$.

$L_{c1-reg}^{c}$ and $L_{c2-reg}^{c}$ are regression losses for the box center predictions, $L_{r-cls}^{c}$ and $L_{r-reg}^{c}$ are the rotation anchor classification and regression losses, $L_{s-cls}^{c}$ and $L_{s-reg}^{c}$ are the size anchor classification and regression losses, and $L_{corner}^{c}$ is the regression loss for the 8 vertices of the 3D box. Note that each loss term in $L_{box}^{3D}$ is weighted by $w_{s}$, which we omit in Eq. 2 and all subsequent losses for brevity. Until this point, the network is fully-supervised by 3D box labels available only for classes $C_{3D}$. In the subsequent sections, we describe our contributions to train $f_{box}$ on the weak classes in $C_{2D}$ by leveraging on the BoxPC Fit Network (Sec. 4.2) and weak losses (Sec. 4.3).

4.2 Box to Point Cloud (BoxPC) Fit Network

The initial 3D box predictions $\hat{B}_{0}$ on classes $C_{2D}$ are not reliable because there are no 3D box labels for these classes. As a result, the 3D box surfaces of the initial predictions $\hat{B}_{0}$ for classes $C_{2D}$ are likely to cut the frustum point cloud $X$ at unnatural places, as illustrated on the right of Fig. 3. We utilize our BoxPC Fit Network $f_{boxpc}$ with a novel training method to transfer the knowledge of a good BoxPC fit from the 3D box labels of classes $C_{3D}$ to classes $C_{2D}$. The input to our $f_{boxpc}$ is a pair of 3D box $B$ and frustum point cloud $X$. The outputs of $f_{boxpc}$ are the BoxPC Fit probability, i.e. goodness-of-fit

[TABLE]

between $B$ and the object in $X$, and the correction

[TABLE]

required on $B$ to improve the fit between $B$ and the object in $X$. $\hat{p}\rightarrow 1$ when $B$ encloses the object in $X$ tightly, i.e. there is a high overlap between $B$ and the 3D box label $B^{*}$.

A pretrained $f_{boxpc}$ (see Sec. 4.2.1 for the training procedure) is used to supervise and improve 3D box predictions $\hat{B}$ for classes $C_{2D}$. Specifically, we train $f_{box}$ to make better initial 3D box predictions $\hat{B}_{0}$ by maximizing $\hat{p}$. By minimizing the loss:

[TABLE]

i.e. maximizing $\hat{p}$, our network $f_{box}$ learns to predict $\hat{B}_{0}$ that fit objects in their respective point clouds well. Finally, we obtain the final 3D box prediction $\hat{B}$ by correcting the initial 3D box prediction $\hat{B}_{0}$ with $\Delta\hat{B}$, i.e. $\hat{B}=\hat{B}_{0}+\Delta\hat{B}$.

4.2.1 Pretraining BoxPC Fit Network

We train the BoxPC Fit network $f_{boxpc}$ on the classes in $C_{3D}$ with the 3D box labels $B^{*}\in\mathbb{R}^{7}$ and their corresponding point clouds $X$. We sample varying degree of perturbations $\delta\in\mathbb{R}^{7}$ to the 3D box labels and get perturbed 3D boxes $B^{*}-\delta$. We define 2 sets of perturbations, $\textbf{P}^{+}(B^{*})$ and $\textbf{P}^{-}(B^{*})$, where $\textbf{P}^{+}$ and $\textbf{P}^{-}$ are sets of small and large perturbations, respectively. Formally, $\textbf{P}^{+}(B^{*})=\{\delta:{\alpha}^{+}\leq IOU(B^{*}-\delta,B^{*})\leq{\beta}^{+}\mid\{{\alpha}^{+},{\beta}^{+}\}\in[0,1]\}$, where $IOU$ refers to the 3D Intersection-over-Union. $\textbf{P}^{-}$ is defined similarly but with ${\alpha}^{-}$ and ${\beta}^{-}$ as the bounds. ${\alpha}^{+}$ and ${\beta}^{+}$ are set such that the 3D boxes perturbed by $\delta\in\textbf{P}^{+}$ have high overlaps with $B^{*}$ or good fit with its point clouds and vice versa for $\textbf{P}^{-}$. The loss function to train $f_{boxpc}$ is given by

[TABLE]

where $L_{cls}^{c}$ is the classification loss

[TABLE]

to predict the fit of a perturbed 3D box and its frustum point cloud, and $L_{reg}$ is the regression loss

[TABLE]

to predict the perturbation to correct the perturbed 3D box. $H$ is the binary cross-entropy and $l(y^{*},\hat{y})$ is the Smooth L1 loss with $y^{*}$ as the target for $\hat{y}$.

4.2.2 BoxPC Feature Representations

A possible way to encode the feature respresentations for the BoxPC network is illustrated on the left of Fig. 4. The features of the input 3D box and frustum point cloud are learned using several MLPs and PointNets [22], respectively. These features are then concatenated and fed into several layers of MLPs to get the BoxPC fit probability $\hat{p}$ and Box Correction term $\Delta\hat{B}$. However, this neglects the intricate information of how the 3D box surfaces cut the point cloud, leading to poor performance as shown in our ablation studies in Sec. 6.3. Hence, we propose another method to combine the 3D box and point cloud in the input space in a differentiable manner that also exploits the relationship between them. Specifically, we use the box center $(b_{x},b_{y},b_{z})$ to translate the point cloud to the origin to enhance translational invariance to the center positions. Next, we use the box sizes $(b_{h},b_{w},b_{l})$ and rotation $b_{\theta}$ to form six planes representing each of the six box surfaces. We compute a $N\times 6$ feature vector of the perpendicular distance (with direction) from each point to each of the six planes. This feature vector is concatenated with the original frustum point cloud $X\in\mathbb{R}^{N\times(3+k)}$ to give the box-combined frustum point cloud $X^{\prime}\in\mathbb{R}^{N\times(9+k)}$ as illustrated on the right of Fig. 4. This representation allows the network to easily reason about regions where the box surfaces cut the point cloud since the perpendicular distance will be close to 0. Additionally, it is possible to perform feature learning on the 3D box and the point cloud jointly with a single PointNet-based [22] network, improving performance.

4.3 Weak Losses

We supervise the initial 3D box predictions $\hat{B}_{0}$ with 2D box labels $B^{*}_{2D}$ and additional priors.

4.3.1 Relaxed Reprojection Loss

Despite the correlation, the 2D box that encloses all the points of a projected 3D box label does not coincide with the 2D box label. We show in Sec. 6.3 that performance deteriorates if we simply minimize the reprojection loss between the 2D box label and the 2D box enclosing the projection of the predicted 3D box onto the image plane. Instead, we propose a relaxed reprojection loss, where boundaries close to the 2D box labels are not penalized. Let $B^{*}_{2D}=[b^{*}_{left},b^{*}_{top},b^{*}_{right},b^{*}_{bot}]$ (green box in Fig. 5) that consists of the left, top, right and bottom image coordinates be the 2D box label with the top-left hand corner as origin. We define an upper bound box $B^{*}_{upper}=B^{*}_{2D}+\mathcal{U}$ and a lower bound box $B^{*}_{lower}=B^{*}_{2D}+\mathcal{L}$ (outer and inner blue boxes in Fig. 5), where $\mathcal{U}=[u_{1},u_{2},u_{3},u_{4}]$ and $\mathcal{L}=[v_{1},v_{2},v_{3},v_{4}]$ are vectors used to adjust the size of $B^{*}_{2D}$. Given an initial 3D box prediction $\hat{B}_{0}$, we project it onto the image plane $\pi(\hat{B}_{0})$ (red box in Fig. 5) and obtain an enclosing 2D box around the projected points $\hat{B}_{0}^{reproj}=g(\pi(\hat{B}_{0}))$ (yellow box in Fig. 5), where $g(.)$ is the function that returns the enclosing 2D box. We penalize $\hat{B}_{0}^{reproj}$ for violating the bounds specified by $B^{*}_{upper}$ and $B^{*}_{lower}$. An example is shown in Fig. 5, the bounds on the left and right are not violated because the left and right sides of the 2D box $\hat{B}_{0}^{reproj}$ stays within the bounds specified by $B^{*}_{upper}$ and $B^{*}_{lower}$ (blue boxes). However, the top and bottom of $\hat{B}_{0}^{reproj}$ beyond the bounds are penalized. More formally, the relaxed reprojection loss is given by

[TABLE]

$\tilde{l}(y^{*}_{lower},y^{*}_{upper},\hat{y})$ is a relaxed Smooth L1 loss such that there is no penalty on $\hat{y}$ if $y^{*}_{lower}\leq\hat{y}\leq y^{*}_{upper}$, and $\hat{B}_{0}^{c,reproj}(i)$ retrieves the $i$-th component of $\hat{B}_{0}^{c,reproj}$.

4.3.2 Box Prior Loss

We use prior knowledge on the object volume and size to regularize the training loss. More specifically, we add the prior loss

[TABLE]

to train our network, where

[TABLE]

is to penalize predictions with volumes that are below class-specific thresholds from prior knowledge about the scale of objects, and

[TABLE]

is to penalize the size variance of the predictions within each class in each minibatch since there should not be excessive size variations within a class. $(\hat{b}_{0,h}^{\,c},\hat{b}_{0,w}^{\,c},\hat{b}_{0,l}^{\,c})$ are the size predictions from $\hat{B}_{0}$ for class $c$, and $(\bar{b}_{0,h}^{\,c},\bar{b}_{0,w}^{\,c},\bar{b}_{0,l}^{\,c})$ are the average size predictions per minibatch for class $c$ and $V^{c}$ is the volume threshold for class $c$.

5 Training

We train a Faster RCNN-based [23] 2D object detector with the 2D box labels from classes $C$ and train the BoxPC Fit Network by minimizing $L_{boxpc}^{3D}$ from Eq. 6 with the 3D box labels from classes $C_{3D}$. Finally, when we train the 3D object detector, we alternate between optimizing the losses for classes in $C_{2D}$ and classes in $C_{3D}$. Specifically, when optimizing for $C_{2D}$, we train $f_{box}$ to minimize

[TABLE]

where $L_{fit}^{2D}$, $L_{reproj}^{2D}$ and $L_{prior}^{2D}$ are from Eq. 5, 9 and 11 respectively. When optimizing for $C_{3D}$, we train $f_{seg}$ and $f_{box}$ to minimize $L_{seg}^{3D}$ and $L_{box}^{3D}$ from Eq. 1 and 2 respectively. Hence, the loss functions for $f_{seg}$ and $f_{box}$ are respectively given by

[TABLE]

6 Experiments

6.1 Datasets

We evaluate our model on SUN-RGBD and KITTI benchmarks for 3D object detection. For the SUN-RGBD benchmark, we evaluate on 10 classes as in [21]. To test the performance of cross-category semi-supervised 3D object detection (CS3D), we randomly split the 10 classes into 2 subsets ($C_{A}$ and $C_{B}$) of 5 classes each (the 5 classes on the left and on the right of Tab. 1). First, we let $C_{A}$ be the strong classes and $C_{B}$ be the weak classes by setting $C_{3D}=C_{A},C_{2D}=C_{B}$ to obtain the evaluation results for $C_{B}$. Next, we reversed $C_{A}$ and $C_{B}$ to obtain the evaluation results for $C_{A}$. We follow the same train/val split and performance measure of average precision (AP) with 3D Intersection-over-Union (IoU) of 0.25 as in [21].

The KITTI benchmark evaluates on Cars, Pedestrians and Cyclists classes. To test the CS3D setting, we set any 2 classes to $C_{3D}$ and the remaining class to $C_{2D}$ to obtain the evaluation results on $C_{2D}$. For example, to obtain the evaluation results on Cyclists, we set $C_{3D}=\text{ \{{Cars}, {Pedestrians}\} }$. We follow the same train/val split as in [21] and measure performance using AP with 3D IoU of 0.25 for all classes. For fully-supervised setting, we use 3D IoU of 0.7 for Cars and 0.5 for Pedestrians and Cyclists.

6.2 Comparison with Baselines

To the best of our knowledge, there are no other methods that demonstrate weakly- or semi-supervised 3D object detection, therefore we design baselines (details in supplementary) with the state-of-the-art FPN [21] where it trains on strong labels from $C_{3D}$ and tests on $C_{2D}$. Specifically, we use the original network (“FPN*”) and a network without the one hot class vector (“FPN* w/o OneHot”). The latter performs better since there are no 3D box labels for classes $C_{2D}$. Finally, we train fully-supervised BoxPC Fit networks to further improve the fully-supervised FPN [21].

SUN-RGBD

In Tab. 1, BoxPC, R, P refer to the usage of the proposed BoxPC Fit network, relaxed reprojection loss and box prior loss. The fully-supervised “FPN*” serves as an upper-bound performance for the semi-supervised methods. The semi-supervised baselines “FPN*” and “FPN* w/o OneHot” perform poorly because they are not able to reason about the weak classes and predict boxes that are only roughly correct. By adding R and BoxPC, it allows the network to reason about 3D box predictions on weak classes, improving performance significantly from 30.1% to 41.1%. Finally, the addition of the prior knowledge P allows our model to achieve a good performance of 41.8% mAP, which is 81.2% of the fully-supervised “FPN*” (vs 58.4% for baseline “FPN* w/o OneHot”). We consider this a promising result for semi-supervised methods, and it demonstrates the effectiveness of our proposed model in transferring knowledge to unseen classes. In addition, we show the effectiveness of the BoxPC Fit network in fully-supervised settings by training the BoxPC Fit network on all classes $C$ and using the network to refine the 3D box predictions of the state-of-the-art FPN [21]. As seen in Tab. 1, “FPN* + BoxPC Refine” outperforms the vanilla fully-supervised “FPN*” in every single class except for bookshelf, showing the usefulness of the BoxPC Fit network even in fully-supervised settings.

KITTI

In Tab. 3, the baselines are unable to achieve any performance on Cars when trained on Pedestrians and Cyclists because of huge differences in sizes. The network simply assumes a Car instance is small and makes small predictions. With the addition of a prior volume, it becomes possible to make predictions on Cars. We observe that adding “BoxPC + R + P” significantly improves performance over the baseline’s mean AP of 0.07% to 59.95% for Cars. It also improved the baseline’s performance from 48.03% to 67.49% for Pedestrians and from 41.89% to 51.83% for Cyclists. Similarly, we show that a fully-supervised BoxPC Fit network is able to refine and improve the 3D box predictions made by “FPN” in Tab. 3. In this case, “FPN + BoxPC Refine” improves AP for all classes at all difficulty levels over the original “FPN”.

6.3 Ablation Studies

Relaxed Reprojection Loss

To illustrate the usefulness of using a relaxed version of the reprojection loss, we vary the lower and upper bounds $B^{*}_{lower}$ and $B^{*}_{upper}$. We set $B^{*}_{lower},B^{*}_{upper}$ to different scales of $B^{*}_{2D}$, i.e $1.5\ B^{*}_{2D}$ refers to the 2D box centered at the same point as $B^{*}_{2D}$ but with 1.5$\times$ size. By setting $B^{*}_{lower},B^{*}_{upper}=B^{*}_{2D}$, it reverts from the relaxed version to the direct version of reprojection loss. We train with similar settings as “Ours + R” in Tab. 1, and show in Tab. 5 that mean AP improves from 25.6% ($B^{*}_{upper}=B^{*}_{2D}$) to 37.2% ($B^{*}_{upper}=1.5\ B^{*}_{2D}$).

BoxPC Fit Network Objectives

The BoxPC Fit network is trained on classification and regression objectives derived from the 3D box labels. To show the effectiveness of having both objectives, we adopt different settings in which classification or regression objectives are active. We train with similar settings as “Ours + BoxPC” in Tab. 1. Without the classification objective for the BoxPC Fit network, the 3D object detector is unable to maximize $\hat{p}=f_{boxpc-cls}(X,\hat{B}_{0})$ during training. Omission of the regression objective prevents the initial box predictions $\hat{B}_{0}$ from being refined further by $\Delta\hat{B}=f_{boxpc-reg}(X,\hat{B}_{0})$ to the final box predictions $\hat{B}$. From Tab. 5, we see that training without either classification or regression objective causes performance to drop from 40.6% to 32.1% or 39.8%.

BoxPC Representation

To demonstrate the importance of a joint Box to Point Cloud input representation as seen on the right of Fig. 4, we train on the “Ours + BoxPC” setting in Tab. 1 with different BoxPC representations. As shown in Tab. 5, the performance of combined learning of BoxPC features exceeds the performance of independent learning of BoxPC features by 6.2%.

6.4 Qualitative Results

We visualize some of the predictions by the baseline, fully-supervised and proposed models in Fig. 6. We used 2D box labels for frustum proposals to reduce noise due to 2D detections. We observe that the proposed model’s predictions are closer to the fully-supervised model than the baseline’s. The baseline’s predictions tend to be inaccurate since it is not trained to fit the objects from weak classes.

7 Conclusion

We propose a transferable semi-supervised model that is able to perform 3D object detection on weak classes with only 2D box labels. We achieve strong performance over the baseline approaches and in the fully-supervised setting, we improved the performance of existing detectors. In conclusion, our method improves the practicality and usefulness of 3D object detectors in new applications.

Supplementary Materials

In Sec. A, we discuss the costs of different types of labels. In Sec. B, we observe the performance with varying amounts of 3D box labels for $C_{2D}$ classes. In Sec. C and Sec. D, we elaborate on the details of the networks and the details of the training procedures, respectively. In Sec. E, we provide additional qualitative results and figures on the SUN-RGBD dataset and in Sec. F, qualitative results for the KITTI dataset.

A Time Costs of Labels

The SUN-RGBD dataset [26] required 2,051 hours to label 64,595 3D bboxes, which is an average of 114s per object. In contrast, the average time to label a 2D bbox according to [18] is 35s per object but can be as fast as 7s when using their proposed labeling method with no loss of accuracy. Hence, it is potentially 3-16 times faster to label 2D compared to 3D bboxes.

B In-Category Semi-supervision Performance

In the main paper, we assumed that there are no 3D box labels for weak classes $C_{2D}$, which is a cross-category semi-supervised setting. In Fig. 7, we train our model and the baseline “FPN*” on varying amounts of 3D box labels for $C_{2D}$ to understand the performance of our model in an in-category semi-supervised setting. When the percentage of 3D box labels used for $C_{2D}$ is $100\%$, the semi-supervised baseline “FPN*” (green line) becomes the fully-supervised “FPN*” (51.5% mAP) and our proposed semi-supervised method (blue line) becomes the fully-supervised “FPN* + BoxPC Refine” (53.2% mAP).

We observe that our proposed method always performs better than the baseline for different percentages of 3D box labels of $C_{2D}$ classes. This demonstrates the usefulness of our method even when labels are available for $C_{2D}$ classes. Additionally, we note that the baseline with 50% 3D box labels available for $C_{2D}$ achieves a similar performance to our proposed method with 0% 3D box labels for $C_{2D}$, which demonstrates the effectiveness of the knowledge that has been transferred from $C_{3D}$ classes.

C Network Details

C.1 Network Architecture of Baselines

Fig. 9 shows the network architecture for the original Frustum PointNets (FPN) [21]. The same architecture is used to obtain the results for “FPN” and the baseline “FPN*” in Tab. 1 of the main paper. We remove the one hot class vectors given as features to the $f_{seg}$ and $f_{box}$ networks to get the stronger baseline “FPN* w/o OneHot”. The performance improves because in the cross-category semi-supervised setting, the network does not train on the strong labels of inference classes. Hence, having class information does not help the network during inference.

C.2 Network Component Details

In this section, we describe the details of the network components in the baseline and proposed models for the cross-category semi-supervised 3D object detection (CS3D) setting. The fully-supervised 3D object detection (FS3D) setting uses the same components except for minor changes.

The network components are given in Fig. 9. We use the v1 instance segmentation and v1 box estimation networks in FPN [21] as the instance segmentation and box estimation networks in our baseline and proposed models. The BoxPC Fit networks also use PointNet [22] and MLP layers to learn the BoxPC features.

In the CS3D setting, we remove the one hot class vector that is originally concatenated with the “Global Features” of the v1 instance segmentation network because we are performing class-agnostic instance segmentations. In the FS3D setting, the one hot class vector is added back.

The v1 box estimation network is composed of the T-Net and Box Estimation PointNet. The T-Net gives an initial prediction for the center of the box $b_{x},b_{y},b_{z}$ which is used to translate the input point cloud to reduce translational variance. Then, the Box Estimation PointNet makes the box predictions using the translated point cloud. Both CS3D and FS3D settings use the same box estimation networks. The details of the loss functions to train the box estimation networks can be found in [21].

In the CS3D setting, the BoxPC Fit network learns class-agnostic BoxPC Fit between 3D boxes and point clouds. In the FS3D setting, we concatenate a one hot class vector to the “BoxPC Features” to allow the network to learn BoxPC Fit that is specific to each class.

D Training Details

D.1 Training of BoxPC Fit Network

As discussed in the paper, we have to sample from 2 sets of perturbations $\textbf{P}^{+}$ and $\textbf{P}^{-}$ to train the BoxPC Fit Network to understand what is a good BoxPC fit. To sample perturbations $\delta=[\delta_{x},\delta_{y},\delta_{z},\delta_{h},\delta_{w},\delta_{l},\delta_{\theta}]$ from either $\textbf{P}^{+}$ or $\textbf{P}^{-}$, we uniformly sample center perturbations $\delta_{x},\delta_{y},\delta_{z}\in[-0.8,0.8]$, size perturbations $\delta_{h},\delta_{w},\delta_{l}\in[-0.2,0.2]$ and rotation perturbations $\delta_{\theta}\in[0,\pi]$. We perturb a 3D box label $B^{*}$ to obtain a perturbed 3D box label $B^{*}-\delta$ with a sampled perturbation $\delta$. Next, we check if $B^{*}-\delta$ has an IOU with $B^{*}$ that is within the range specified by the set $\textbf{P}^{+}$ or $\textbf{P}^{-}$, i.e if ${\alpha}^{+}\leq IOU(B^{*}-\delta,B^{*})\leq{\beta}^{+}$ or ${\alpha}^{-}\leq IOU(B^{*}-\delta,B^{*})\leq{\beta}^{-}$, respectively. We accept and use it as a single input if it satisfies the IOU range for the set. We repeat the process until we have enough samples for a minibatch. Specifically, each minibatch has equal number of $B^{*}-\delta$ samples from $\textbf{P}^{+}$ and $\textbf{P}^{-}$.

D.2 Cross-category Semi-supervised Learning

Let $C_{A}$ = {bathtub, bed, toilet, chair, desk} and $C_{B}$ = {dresser, nightstand, sofa, table, bookshelf}. We train with $C_{3D}=C_{B}$ and $C_{2D}=C_{A}$, i.e. we train on the 2D and 3D box labels of $C_{B}$ and the 2D box labels of $C_{A}$, to obtain the evaluation results for CS3D on $C_{A}$. We train with $C_{3D}=C_{A}$ and $C_{2D}=C_{B}$ to get the evaluation results for CS3D on $C_{B}$.

SUN-RGBD

For our 2D object detector, we train a Faster RCNN [23, 4] network on all classes $C=C_{2D}\cup C_{3D}$. When we train the BoxPC Fit network on classes $C_{3D}$, we set the perturbation parameters to $\alpha^{+}=0.7,\beta^{+}=1.0,\alpha^{-}=0.01,\beta^{-}=0.25$. The loss weights for the BoxPC Fit network are set to $w_{cls}=1,w_{reg}=4$. When we train the 3D object detector, we set the lower and upper bound boxes for the relaxed reprojection loss to $B^{*}_{lower}=B^{*}_{2D},B^{*}_{upper}=1.5\ B^{*}_{2D}$ and volume threshold for all classes to $V=0$. The loss weights for the 3D detector are $w_{c1-reg}=0.1,w_{c2-reg}=0.1,w_{r-cls}=0.1,w_{r-reg}=2,w_{s-cls}=0.1,w_{s-reg}=2,w_{corner}=0.1,w_{fit}=0.05,w_{reproj}=0.0005,w_{vol}=0,w_{s-var}=0.1$.

KITTI

For our 2D object detector, we use the released detections from FPN [21] for fair comparisons with FPN. We set the perturbation parameters to $\alpha^{+}=0.8,\beta^{+}=1.0,\alpha^{-}=0.01,\beta^{-}=0.4$ when we train the BoxPC Fit network on classes $C_{3D}$. The loss weights of the BoxPC Fit network are set to $w_{cls}=1,w_{reg}=4$. We set the lower and upper bound boxes for the relaxed reprojection loss to $B^{*}_{lower}=B^{*}_{2D},B^{*}_{upper}=1.5\ B^{*}_{2D}$ for Pedestrians and Cyclists, and $B^{*}_{lower}=B^{*}_{2D},B^{*}_{upper}=1.25\ B^{*}_{2D}$ for Cars when we train the 3D object detector on the classes $C$. We set the volume threshold for Cars to $V^{car}=10$ and 0 for the other classes. The loss weights for the 3D object detector are set to $w_{c1-reg}=1,w_{c2-reg}=1,w_{r-cls}=1,w_{r-reg}=20,w_{s-cls}=1,w_{s-reg}=20,w_{corner}=10,w_{fit}=1,w_{reproj}=0.2,w_{vol}=1,w_{s-var}=2$.

D.3 Fully-supervised Learning

In the fully-supervised setting, we train and evaluate on $C$.

SUN-RGBD

We use the same 2D detector as in the CS3D setting. When we train our BoxPC Fit network, we set the perturbation parameters to $\alpha^{+}=0.6,\beta^{+}=1.0,\alpha^{-}=0.05,\beta^{-}=0.4$. The loss weights of the BoxPC Fit network are set to $w_{cls}=1,w_{reg}=4$. For our 3D object detector, we set the parameters to $w_{c1-reg}=0.1,w_{c2-reg}=0.1,w_{r-cls}=0.1,w_{r-reg}=2,w_{s-cls}=0.1,w_{s-reg}=2,w_{corner}=0.1$. Next, we use the trained BoxPC Fit network to refine the 3D box predictions of this fully-supervised model without further training.

KITTI

We use the same 2D detections as the CS3D setting. We train one BoxPC Fit network for each of the 3 classes to allow each network to specialize on improving box predictions for a single class. For Cars, we set $\alpha^{+}=0.7,\beta^{+}=1.0,\alpha^{-}=0.6,\beta^{-}=0.8$. For Pedestrians, we set $\alpha^{+}=0.7,\beta^{+}=1.0,\alpha^{-}=0.3,\beta^{-}=0.7$. For Cyclists, we set $\alpha^{+}=0.7,\beta^{+}=1.0,\alpha^{-}=0.5,\beta^{-}=0.7$. The loss weights of the BoxPC Fit network are set to $w_{cls}=0,w_{reg}=4$. Next, we use the trained BoxPC Fit networks to refine the 3D box predictions of the 3D object detector model released by [21] without further training.

E Additional Results for SUN-RGBD

In Fig. 10, we plot the precision-recall curves for different methods to study the importance of each proposed component. All the methods are cross-category semi-supervised except for “FPN (Fully-supervised)”, which is the fully-supervised FPN [21] that gives an upper-bound performance for the methods. We observe that “Ours + BoxPC + R + P” (in yellow) has higher precision at every recall than the baseline “FPN w/o OneHot” (in blue) for almost all classes.

We also provide additional qualitative results on the SUN-RGBD dataset in Fig. 11. The predictions made by the baseline model tend to be large and inaccurate due to the lack of strong labels in the weak classes. In the third example of Fig. 11, we see that the predictions by the baseline model can also be highly unnatural as it cuts through the wall. On the contrary, we observe that the proposed model’s predictions tend to be more reasonable and closer to the fully-supervised model despite not having strong labels for the weak classes. In the last two examples of Fig. 11, the heavy occlusions in the scene makes it difficult for the baseline and proposed models to make good predictions.

F Qualitative Results for KITTI

In Fig. 12 and Fig. 13, we provide qualitative comparisons between the baseline, fully-supervised and proposed models for the KITTI dataset. In both Fig. 12 and Fig. 13, we again observe that the proposed model is closer to the fully-supervised model than the baseline model. Notably, in the first example (top left) and ninth example (bottom right) of Fig. 12, we observe that the proposed model is able to make significantly good predictions on Pedestrians despite the crowded scene.

For almost all of the Car instances, the baseline model makes excessively small predictions because it was trained on Pedestrian and Cyclist classes which are much smaller. Our proposed model is able to make better predictions on Cars but the orientation of the 3D box predictions can be improved further.

In the last two examples of Fig. 13, we show some of the difficult scenes for our baseline and proposed models. This is because there are heavy occlusions and poor understanding of Cars since the models were trained on Pedestrians and Cyclists, which have very different shapes and sizes from Cars.