Shift R-CNN: Deep Monocular 3D Object Detection with Closed-Form Geometric Constraints

TL;DR

Shift R-CNN introduces a hybrid deep learning and geometric approach for monocular 3D object detection, achieving top results on KITTI without pre-trained depth models.

Contribution

It combines a modified Faster R-CNN with a closed-form geometric solution and a novel Volume Displacement Loss for improved monocular 3D detection.

Findings

Achieves state-of-the-art results on KITTI benchmark.

Does not rely on pre-trained depth estimation networks.

Introduces a novel geometric constraint-based refinement.

Abstract

We propose Shift R-CNN, a hybrid model for monocular 3D object detection, which combines deep learning with the power of geometry. We adapt a Faster R-CNN network for regressing initial 2D and 3D object properties and combine it with a least squares solution for the inverse 2D to 3D geometric mapping problem, using the camera projection matrix. The closed-form solution of the mathematical system, along with the initial output of the adapted Faster R-CNN are then passed through a final ShiftNet network that refines the result using our newly proposed Volume Displacement Loss. Our novel, geometrically constrained deep learning approach to monocular 3D object detection obtains top results on KITTI 3D Object Detection Benchmark, being the best among all monocular methods that do not use any pre-trained network for depth estimation.

Figures

Tables

Table 1. Table 1 : 3D object detection results on KITTI Chen/Test splits. We report A P 3 ​ D ( % ) AP_{3D}(\%) and A P B ​ E ​ V ( % ) AP_{BEV}(\%) for Car, Pedestrian and Cyclist classes. Methods that use a pre-trained monocular depth network (Mono+PD) are in blue . We denote with bold black monocular state of the art, bold black * cases when we outperform one Mono+PD method and bold blue , cases where we outperform all Mono+PD. Best viewed in color.

			$A{P}_{3D}$ (%)			$A{P}_{BEV}$ (%)
Method	Setup	Class	Easy	Moderate	Hard	Easy	Moderate	Hard
Mono3D [1]	Mono	Car (IoU$\ge$0.7)	2.53 /       -	2.31 /       -	2.31 /       -	5.22 /       -	5.19 /       -	4.13 /       -
DeepBox3D [16]	Mono		-       /    5.85	-       /    4.10	-       /    3.84	-       /    9.99	-       /    7.71	-       /    5.30
OFT-Net [20]	Mono		4.07 /    2.50	3.27 /    3.28	3.29 /    2.27	11.06 /    9.50	8.79 /    7.99	8.91 /    7.51
MLF-Mono [23]*	Mono+PD		10.53 /    7.85	5.69 /    5.39	5.39 /    4.73	22.03 / 19.20	13.63 / 12.17	11.60 / 10.89
ROI-10D [15]*	Mono+PD		10.25 / 12.30	6.39 / 10.30	6.18 /    9.39	14.76 / 16.77	9.55 / 12.40	7.57 / 11.39
Linear System (Ours)	Mono		7.24 /    6.80	5.98 /    4.14	5.54 /    3.50	14.74 / 11.75	12.48 /    8.34	11.22 /    6.80
ShiftNet (Ours)	Mono		13.84 /     8.13*	11.29 /    5.22	11.08 /    4.78*	18.61* / 13.32	14.71 /    8,49	13.57 /    6.40
OFT-Net [20]	Mono	Ped. (IoU$\ge$0.5)	-       /    1.11	-       /    1.06	-       /    1.06	-       /    1.55	-       /    1.93	-       /    1.65
Linear System (Ours)	Mono		1.51 /    0.53	1.51 /    0.53	1.51 /    0.53	1.51 /    0.53	1.51 /    0.53	1.51 /    0.53
ShiftNet (Ours)	Mono		7.55 / 13.36	6.80 / 10.59	6.12 / 10.59	8.24 / 13.81	7.50 / 11.44	6.73 / 10.76
OFT-Net [20]	Mono	Cyc. (IoU$\ge$0.5)	-       /    0.43	-       /    0.43	-       /    0.43	-       /    0.43	-       /    0.79	-       /    0.43
Linear System (Ours)	Mono		1.38 /    0.73	0.90 /    0.43	0.90 /    0.43	1.42 /    0.53	0.90 /    0.53	0.90 /    0.53
ShiftNet (Ours)	Mono		1.85 /    3.03	1.08 /    3.03	1.10 /    3.03	2.30 /    3.58	2.00 /    3.03	2.11 /    3.03

Equations

$$L_{{\alpha }_L}=||\sin {\alpha }_{L_{gt}}-\widehat{{\sin }_{{\alpha }_L}}|{|}_2+||\cos {\alpha }_{L_{gt}}-\widehat{{\cos }_{{\alpha }_L}}|{|}_2+L_{cnt},$$

$$L_{cnt}=||1-({\widehat{{\sin }_{{\alpha }_L}}}^2+{\widehat{{\cos }_{{\alpha }_L}}}^2)|{|}_2.$$

$$\Delta\mathbf{d}_{c}=\left[{\begin{array}[]{ccc}\ln{\frac{h}{\overline{h_{c}}}},\ln{\frac{w}{\overline{w_{c}}}},\ln{\frac{l}{\overline{l_{c}}}}\end{array}}\right],$$

$$\mathbf{d}=[e^{\Delta h}\times \overline{h_c},e^{\Delta w}\times \overline{w_c},e^{\Delta l}\times \overline{l_c}].$$

$$L_{total}=w_{cls}{L}_{cls}+w_{2D}{L}_{2D}+w_{{\alpha }_L}{L}_{{\alpha }_L}+w_{dim}{L}_{dim}.$$

$${\alpha }_G={\alpha }_L-{\theta }_{ray}.$$

$${\mathbf{x}}_{3D}={\mathbf{R}}_y({\alpha }_G){\mathbf{x}}_{3D_o}+{\mathbf{t}}^{\prime }.$$

$$\lambda\left[\begin{array}[]{c}\mathbf{x}_{2D}\\ 1\end{array}\right]=\mathbf{P}\left[\begin{array}[]{c}\mathbf{x}_{3D}\\ 1\end{array}\right].$$

$$\underbrace{\mathbf{P}\left[\begin{array}[]{cc}I&\mathbf{R}_{y}(\alpha_{G})X_{3D_{o}}\\ 0&1\end{array}\right]}_{\textit{$\mathbf{M}$}}\left[\begin{array}[]{c}\mathbf{t}^{\prime}\\ 1\end{array}\right]=\lambda\left[\begin{array}[]{c}x_{2D_{side}}\\ y_{2D_{side}}\\ 1\end{array}\right].$$

$$\left[\begin{array}[]{c}\mathbf{m}_{1}^{\top}-\mathbf{m}_{3}^{\top}x_{min}\\ \mathbf{m}_{2}^{\top}-\mathbf{m}_{3}^{\top}y_{min}\\ \mathbf{m}_{1}^{\top}-\mathbf{m}_{3}^{\top}x_{max}\\ \mathbf{m}_{2}^{\top}-\mathbf{m}_{3}^{\top}y_{max}\end{array}\right]\mathbf{t}^{\prime}=\left[\begin{array}[]{c}m_{34}x_{min}-m_{14}\\ m_{34}y_{min}-m_{24}\\ m_{34}x_{max}-m_{14}\\ m_{34}y_{max}-m_{24}\end{array}\right].$$

$$\Delta {\mathbf{t}}_{{\alpha }_G}={\mathbf{R}}_y({\alpha }_G)\Delta \mathbf{t}.$$

$$L=\underset{A_{yOz}}{\underbrace{w\times h}}\times |\Delta x_{{\alpha }_G}|+\underset{A_{xOz}}{\underbrace{w\times l}}\times |\Delta y_{{\alpha }_G}|+\underset{A_{xOy}}{\underbrace{h\times l}}\times |\Delta z_{{\alpha }_G}|.$$

Full text

Index Terms—  Monocular 3D object detection, convolutional neural networks, autonomous driving, geometric constraints

1 Introduction

Autonomous driving relies on visual perception of the surrounding environment, in which the ability to perform accurate obstacle detection is a key factor. With many different types of sensors available, state of the art autonomous driving systems are based on sensor fusion [9]. When compared to LiDAR systems, which have a higher cost of production, RGB cameras are ubiquitous, much less expensive, and have a significantly higher pixel resolution. Humans drive using only visual cues, therefore high performance autonomous driving could be achieved, in principle, using vision alone.

While stereo cameras are able to recover the depth of the scene, monocular 3D perception still remains a very difficult challenge. In this paper we focus our efforts on monocular 3D object detection and propose an efficient method that combines the initial Faster R-CNN predictions with a closed-form, 2D to 3D geometric mapping solution that produces a rough estimation of an object’s 3D translation. This translation is then refined with our proposed ShiftNet network.

In summary, the main contributions of our paper are:

A 3D object detection architecture with three stages, that accurately estimates the 2D location, 3D object properties (height, width, length), its orientation and 3D translation in camera coordinates, from a single RGB image (Fig. 1). In the first stage, an adapted Faster R-CNN estimates the 2D box, dimension and orientation. In the second stage, a mathematical system of equations is solved using least squares, based on the previous estimates and the known camera projection matrix, to efficiently compute the 3D object translation by enforcing the 3D to perfectly fit the 2D. In the third stage, a fully connected network, namely ShiftNet, learns the error dependency between the first 2 stages and corrects the final translation. 2. 2.

A novel Volume Displacement Loss (VDL) used to train a network for the task of maximizing the 3D IoU by means of regression. 3. 3.

State of the art results on the KITTI 3D Object Detection Benchmark [5], compared to all other published monocular methods that do not make use of an already trained depth net.

Related Work. There is a vast literature on 2D object detection, with accurate approaches on challenging datasets. They vary from general ones, such as COCO [12], to those dedicated to autonomous driving: KITTI [5], Baidu Apollo [8], and Berkley DeepDrive [24]. 2D object detection architectures can be divided into ROI-based, two-stage models and proposal free, single-shot approaches. Single-shot detectors include YOLO [18], SSD [14] and RetinaNet [11]. Two-stage detectors include Faster R-CNN [19], Mask R-CNN [6], PANet [13].

The most powerful 3D object detection models rely on LiDAR-generated point clouds, which simplify the 3D estimation task. 3D-FCN [10] and VoxelNet [25] apply 3D convolutions to point-cloud data to generate 3D bounding boxes. AVOD [9] uses sensor fusion and jointly predicts 3D properties from LiDAR and RGB, being robust to noisy LiDAR data. PointRCNN [21] performs a point cloud foreground/background segmentation, then pools semantic point clouds for accurate canonical 3D object detection. Other approaches use stereo images for a good trade-off between quality and price. Top methods in this category use depth generated from stereo as well as RGB input. Methods include [2, 3, 17], which use stereo information to create 3D object proposals that are scored with CNNs.

Monocular 3D object detection is the most difficult task since it requires tackling the inverse geometrical problem of mapping 2D space into 3D space, in the absence of any true 3D information. Thus, top approaches rely on extra training data in order to make informed 3D estimations. Wang et al. [22] use monocular depth perception networks such as DORN [4] to generate pseudo-point clouds and then apply a state of the art LiDAR-based model [9]. ROI-10D [15] uses monocular depth estimation and lifts 2D detection into a 6D pose problem. Mono3D [1] enforces 3D candidates to lay on the ground plane, orthogonal to image plane. It also uses semantic segmentation, contextual information, object size, shape and location priors. Deep3DBox [16] uses geometric constraints by tightly fitting the 3D bounding box into the 2D box. Xu et al. [23] use multi-level fusion of Faster R-CNN and depth features for 3D object detection. Advances in stereo and monocular depth estimation could provide accurate 3D information, which could greatly improve non-LiDAR based 3D object detection systems.

2 Our 3D Object Detection Architecture

In the absence of any true depth information, estimating the 3D object translation from a single image is a harder problem than estimating its 2D bounding box, dimension and orientation. Our Shift R-CNN model (Fig. 1), combining deep learning with geometric constraints is mainly based on this observation.

2.1 Stage 1: 2D Object Detection and 3D Intrinsics Estimation

The first stage augments the Faster R-CNN model with additional learning objectives. First, we extract and score 2D region proposals by means of 2D anchors [19], using objectness classification and 2D box regression. We then employ 2D ROI pooling for feature cropping. Based on the top scoring 2D proposals, we refine the cropped features using a convolutional encoder, then split them up into 4 separate heads. For the 2D part we use standard multi-class classification and 2D box refinement. The additional 2 heads that we introduce handle local orientation and 3D dimension regression.

Angle Regression Head. Due to the periodic nature of angles, it is harder to regress them explicitly. Thus, we regress instead the estimates $\widehat{\sin_{\alpha_{L}}}$ and $\widehat{\cos_{\alpha_{L}}}$ of the local orientation angle $\alpha_{L}$ w.r.t the y-axis, by also adding a constraint loss to enforce $\widehat{\sin_{\alpha_{L}}}^{2}+\widehat{\cos_{\alpha_{L}}}^{2}=1$. We predict for each object $C$ different angles, one for each class, and compute the loss only on the winning one w.r.t the ground-truth local angle $\alpha_{L_{gt}}$.

[TABLE]

[TABLE]

At inference time, $\alpha_{L}=atan2(\widehat{\sin_{\alpha_{L}}},\widehat{\cos_{\alpha_{L}}})$.

3D Dimension Regression Head. The 3D object height, width and length (in meters) are also regressed in $C$ triplets, one per class. We do not regress the absolute dimensions, due to their low variance per class. We adopt a procedure similar to 2D bounding box regression [19], by taking into account predefined mean dimensions. Let the mean dimension for a particular class $c$ be \overline{\mathbf{d}_{c}}=\left[{\begin{array}[]{ccc}\overline{h_{c}},\overline{w_{c}},\overline{l_{c}}\\ \end{array}}\right], we denote $\mathbf{d}=[h,w,l]$ as the object’s real predicted dimension and regress the logarithmic scale offsets $\Delta\mathbf{d}_{c}=[\Delta h,\Delta w,\Delta l]$ w.r.t $\overline{\mathbf{d}_{c}}$. The relation between $\Delta\mathbf{d}_{c}$ and $\mathbf{d}$ is seen in Eq. 4. The loss is computed only for the winner class.

[TABLE]

[TABLE]

Multi-Task Loss. We present our total Stage 1 loss in Eq. 5. The first 2 terms are the standard Faster R-CNN multiclass loss and 2D bounding box losses. For the 3D dimension regression we use a standard $L_{2}$ loss, $L_{dim}$, over the logarithmic scale offsets $\Delta\mathbf{d}_{c}$.

[TABLE]

2.2 Stage 2: Estimating the 3D Object Translation

Using the 2D bounding box $\mathbf{b}_{2D}=[x_{min},y_{min},x_{max},y_{max}]$, the 3D dimension estimates $\mathbf{d}=[h,w,l]$ and the local orientation angle about the y-axis $\alpha_{L}$, the 3D bounding box reconstruction follows.

From local to global orientation. Having already predicted the local orientation angle $\alpha_{L}$, the global orientation angle $\alpha_{G}$ is then determined by considering the ray from the camera origin to the center of the object $\theta_{ray}$:

[TABLE]

Inverse geometry problem. In order to accurately reconstruct the 3D bounding box, we need to further estimate the real 3D translation vector $\mathbf{t}=[t_{x},t_{y},t_{z}]^{\top}$ of the object center, in 3D camera coordinates. We formulate this estimation in a closed-form, as a least squares solution given by fitting the geometric constraints imposed by the camera projection matrix $\mathbf{P}$. Thus, an object described by $\mathbf{b}_{2D}$, $\mathbf{d}$ and $\alpha_{G}$, will have a depth-constrained translation $\mathbf{t}^{\prime}=[t^{\prime}_{x},t^{\prime}_{y},t^{\prime}_{z}]^{\top}$ in camera coordinates.

To enforce the 3D bounding box projection to fit tightly into the predicted 2D bounding box, we constrain 2 of the 4 vertical 3D edges to lay on a 2D vertical side and the upper and lower 3D corners to lay on a horizontal 2D side. Assuming that objects lay on the ground plane and that by fixing one vertical 3D edge the second vertical one must be diagonally opposed, we have $4\times 1\times 4\times 4=64$ configurations - from which we choose the best fit. We reconstruct the 3D bounding box $\mathbf{x}_{3D_{o}}$ at the camera center, by using the 3D dimensions $\mathbf{d}$, then obtain the final 3D box, in camera coordinates, by applying the rotation about the y-axis $\mathbf{R}_{y}(\alpha_{G})$ and translation $\mathbf{t}^{\prime}$:

[TABLE]

The relation between a 3D point in the world $\mathbf{x}_{3D}=[x,y,z]^{\top}$ and its 2D projection in the image $\mathbf{x}_{2D}=[x,y]^{\top}$, using the projection matrix $\mathbf{P}$, is given in homogeneous coordinates:

[TABLE]

From Eq. 7 and 8, similar to [16], we obtain the following system where translation $\mathbf{t^{\prime}}$ is the unknown vector:

[TABLE]

By substituting each element of $\mathbf{b}_{2D}$, corresponding to a 2D side, and also $\lambda$ into System 9, we propose a least squares solution for the translation, different from [16]. Here $\mathbf{m_{i}}^{\top}=[m_{i1},m_{i2},m_{i3}]$ and $m_{ij}$ is the $(i,j)$ element of matrix $\mathbf{M}$.

[TABLE]

The over-constrained System 10 can be rewritten as $\mathbf{A}\mathbf{t}^{\prime}=\mathbf{b},\mathbf{b}\neq\mathbf{0}$, with a general closed-form solution for the 3D object translation $\mathbf{t}^{\prime}=(\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{b}$. For choosing the best fitting solution, we evaluate the 2D IoU score of the initial 2D and re-projected 3D box.

2.3 Stage 3: Refining the 3D Object Translation

Shift Network. The $\mathbf{t}^{\prime}=[t^{\prime}_{x},t^{\prime}_{y},t^{\prime}_{z}]^{\top}$ translation estimate from Eq. 10 alone is a good approximation of the object 3D location. However, it assumes a perfect mathematical model and, implicitly, perfect Faster R-CNN predictions. Our tests indicate that the estimate is not accurate when predictions are noisy, going from 80% accuracy with a 3D IoU of at least 0.7, to a roughly 14% if noise is present. Even a small error in the predicted values needed to solve Eq. 10, produce huge displacements of $\mathbf{t}^{\prime}$ and thus low 3D IoU score.

To address this limitation, we propose an effective approach for refining the estimate $\mathbf{t}^{\prime}$. We introduce ShiftNet (SHN) in Stage 3. We model ShiftNet as a hybrid that uses the $\mathbf{t}^{\prime}$ estimate from Stage 2 (Eq. 9), with the initial Stage 1 predictions and outputs an improved final translation $\mathbf{t}^{\prime\prime}=[t^{\prime\prime}_{x},t^{\prime\prime}_{y},t^{\prime\prime}_{z}]$. It learns to combine, in complementary ways, the best of the exact geometric universe with the ability of a neural net to learn in the presence of noise. The actual input fed to the network is given by $\mathbf{t}^{\prime}$, $\mathbf{b}_{2D}$, $\mathbf{d}$, $(sin(\alpha_{L}),cos(\alpha_{L}))$, $(sin(\alpha_{G}),cos(\alpha_{G}))$ and $\mathbf{P}$. SHN is then trained to regress $\mathbf{t}^{\prime\prime}$. The architecture has 2 fully-connected layers with 1024 neurons, and a final layer that regresses 3 values for $\mathbf{t}^{\prime\prime}=[t^{\prime\prime}_{x},t^{\prime\prime}_{y},t^{\prime\prime}_{z}]$.

Volume Displacement Loss (VDL). Using an independent component-wise regression of $\mathbf{t}^{\prime\prime}$ does not capture correctly the final IoU detection measure. It does not take into account the object shape variations along the different dimensions. For example, an error along the Y-axis could produce a larger error in total volume than one, by the same amount, along the Z-axis. A better loss is proposed by [15], by summing the Euclidean distances between each pair of the corresponding 3D bounding box corners, but without taking in consideration the different effects errors along the different axes might have, in the final IoU.

Based on this observation, we introduce the Volume Displacement Loss (VDL) to regress $\mathbf{t}^{\prime\prime}$ as the translation that optimizes the IoU score between two 3D boxes reconstructed using the same $\mathbf{d}$ and $\alpha_{G}$ parameters predicted in Stage 1, but one with the the ground-truth $\mathbf{t}$ and the other one, using the estimated $\mathbf{t}^{\prime\prime}$. While the direct, true 3D IoU w.r.t the ground truth bounding box is not differentiable and does not define a distance in metric space, we show that our VDL loss factorizes into a sum of simple terms and can be effectively used in training.

We first express the Signed Translation Displacement Error (STDE) vector $\Delta\mathbf{t}=[t^{\prime\prime}_{x}-t_{x},t^{\prime\prime}_{y}-t_{y},t^{\prime\prime}_{z}-t_{z}]^{\top}$ along the camera coordinates axis. We factor in the rotation $\alpha_{G}$ and decompose the STDE relative to the object’s local coordinate axis, as it can be seen in the following Equation 11:

[TABLE]

The the VDL loss measures the displaced quantity of volume, when moving the object’s center alongside one local coordinate axis. This quantity can be expressed as a function of the 3D bounding box dimensions and elements of $\Delta\mathbf{t}_{\alpha_{G}}=[\Delta x_{\alpha_{G}},\Delta y_{\alpha_{G}},\Delta z_{\alpha_{G}}]^{\top}$. The following Equation 12 defines our Volume Displacement Loss:

[TABLE]

3 Experiments

In this section, we present the experimental setup, compare to state of the art monocular approaches, and test experimentally our contributions.

3.1 Experimental Setup

Dataset. We evaluate our architecture using the KITTI 3D Object Detection dataset [5]. We use the train-val split proposed by Chen et al. [1], which splits the KITTI training set into 3712 training images and 3769 validation images and also evaluate on the online test split. Stage 1 Setup. We use a Faster R-CNN architecture with ResNet-101 backbone, and the additional angle and 3D dimension heads as described in Section 2.1. For ROI pooling we use image features from stride $8$ by atrous convolutions instead of strided ones in the final encoder part. Stage 1 network is trained end-to-end. At inference we use NMS for 2D boxes, angle, and dimension predictions.

Stage 1 Training Procedure. We train the model with SGD for 30 epochs with a batch size of 1, momentum of 0.9, initial learning rate of $3\times 10^{-4}$ and 2 learning rate decays of 0.1 at 1/2 and 2/3 of the epoch count. Faster R-CNN weights are pre-trained on the COCO 2D detection [12] and for the extra heads we use He initialization [7]. Our loss weighting strategy takes into account the uncertainty of a task: 1.0, 2.0, 5.0 and 100.0 for the classification, localization, orientation and dimension regression, respectively. We train the Stage 1 model on the KITTI dataset, using Chen’s split [1] and the entire KITTI training data for the official submission. We train on the Car, Pedestrian, and Cyclist classes. We use random flip data augmentation.

Stage 3 Setup. We pre-train ShiftNet by using the corresponding ground-truth, instead of the Stage 1 generated data. This gives the network a good initialization on how to map ground truth 2D to 3D information. Then we fine-tune the network on real estimates produced by Stage 1. Compared to the Stage 2 least squares solution given by the mathematical system, ShiftNet improves on the 3D detection accuracy (0.7 3D IoU) on Car by 3% on ground truth data (71% vs. 68%) and by 6.6% on real images estimates given by the Faster R-CNN Stage 1 (13.8% vs. 7.2%). Our experiments show that a relatively simple neural net, such as ShiftNet, can infer 3D information even without directly using visual features, as can be seen in Fig. 2. We experiment with augmenting the ground-truth with additional synthetic data, obtained by randomly perturbing the ground truth 3D boxes and then projecting them with the given camera matrix, to obtain 2D boxes information. The additional pre-training on synthetic data significantly improves the accuracy (0.5 3D IoU) for the classes with fewer examples (22% vs. 20% for Pedestrian, 73% vs. 47% for Cyclist), when training on ground-truth input.

Stage 1 and 3 models (Fig. 1) are trained on a NVIDIA GTX 1080 Ti GPU and for one image, the total inference time is 259ms.

3.2 Comparison to other Methods

Our framework is evaluated on the KITTI $AP_{BEV}$ and $AP_{3D}$ evaluation metrics. We use all 3 classes: Car, Pedestrian, Cyclist. Each of the classes is split into 3 difficulties: Easy, Medium, and Hard. In Table 1, we compare with monocular 3D object detection methods. We show in blue methods that use heavier monocular depth estimation networks. As we can see in Table 1, it is clear that our method outperforms all monocular 3D object detection methods that do not use depth estimation in $AP_{3D}$ and $AP_{BEV}$ on both val/test splits for Car class. We outperform OFT-NET [20], which uses orthographic feature transform. We also outperform some methods that use monocular pre-trained depth prediction such as MLF-MONO [23] and ROI-10D [15] on val split, and come close to MLF-MONO on test split. On KITTI test split, these methods improve their results due to the much more training data available for depth prediction. We are also overall state of the art on the Pedestrian and Cyclist classes for monocular 3D detection. Thus, our result contradicts the assumption that these classes are too difficult to predict from monocular data.

4 Discussion and Conclusions

Compared to 2D detection (91%) and object orientation (90%), our $AP_{3D}$ is lower, due to noise in depth prediction, which, for an error of 1 meter, can move the 3D bounding box outside the required 0.7 3D IoU overlap. Overall, our approach proves very competitive, with top results in its category on the KITTI 3D Benchmark. Our framework builds upon the robust Faster R-CNN, augmented with 3D object dimensions and local orientation prediction. We use the output of Faster R-CNN alongside the camera projection matrix to compute an analytic solution to the 3D translation problem. Our geometric solution is then refined with ShiftNet, which learns to leverage the output of Faster R-CNN with the rigorous mathematical solution. By combining the optimal, but less robust analytic solution with the imperfect, but much more robust deep learning model, we provide a hybrid system that takes advantage of the best of both worlds. With a much lighter architecture and without a pre-trained depth network we obtain state of the art results not only on the Car class, but also on the Pedestrian and Cyclist classes, on which we outperform our competitor OFT-NET [20] on all difficulty levels.