3D Packing for Self-Supervised Monocular Depth Estimation

TL;DR

This paper introduces PackNet, a novel self-supervised deep network for monocular depth estimation that leverages 3D convolutions and symmetry, outperforming existing methods on benchmarks and generalizing well without large-scale pretraining.

Contribution

The work presents a new architecture, PackNet, with symmetrical packing and unpacking blocks, enabling effective 3D feature learning for monocular depth estimation without supervised pretraining.

Findings

Outperforms other self, semi, and fully supervised methods on KITTI.

Generalizes better to out-of-domain data like NuScenes.

Operates in real-time without large-scale pretraining.

Abstract

Although cameras are ubiquitous, robotic platforms typically rely on active sensors like LiDAR for direct 3D perception. In this work, we propose a novel self-supervised monocular depth estimation method combining geometry with a new deep network, PackNet, learned only from unlabeled monocular videos. Our architecture leverages novel symmetrical packing and unpacking blocks to jointly learn to compress and decompress detail-preserving representations using 3D convolutions. Although self-supervised, our method outperforms other self, semi, and fully supervised methods on the KITTI benchmark. The 3D inductive bias in PackNet enables it to scale with input resolution and number of parameters without overfitting, generalizing better on out-of-domain data such as the NuScenes dataset. Furthermore, it does not require large-scale supervised pretraining on ImageNet and can run in real-time. Finally, we release DDAD (Dense Depth for Automated Driving), a new urban driving dataset with more challenging and accurate depth evaluation, thanks to longer-range and denser ground-truth depth generated from high-density LiDARs mounted on a fleet of self-driving cars operating world-wide.

Tables

Table 1. Table 1: Average Absolute Trajectory Error (ATE) in meters on the KITTI Odometry Benchmark  [ geiger2013vision ] : All methods are trained on Sequences 00-08 and evaluated on Sequences 09-10. The ATE numbers are averaged over all overlapping 5-frame snippets in the test sequences. M+v refers to velocity supervision (v) in addition to monocular images (M). The GT checkmark indicates the use of ground-truth translation to scale the estimates at test-time.

Method	Supervision	Resolution	GT	Seq. 09	Seq. 10
SfMLearner (Zhou et al. [zhou2017unsupervised])	M	416 x 128	✓	0.021 $\pm$ 0.017	0.020 $\pm$ 0.015
Monodepth2 (Godard et al. [monodepth2])	M	640 x 192	✓	0.017 $\pm$ 0.008	0.015 $\pm$ 0.010
DF-Net (Zou et al. [zou2018dfnet])	M	576 x 160	✓	0.017 $\pm$ 0.007	0.015 $\pm$ 0.009
Vid2Depth (Mahjourian et al. [mahjourian2018unsupervised])	M	416 x 128	✓	0.013 $\pm$ 0.010	0.012 $\pm$ 0.011
GeoNet (Yin et al. [yin2018geonet])	M	416 x 128	✓	0.012 $\pm$ 0.007	0.012 $\pm$ 0.009
Struct2Depth (Casser et al. [casser2018depth])	M	416 x 128	✓	0.011 $\pm$ 0.006	0.011 $\pm$ 0.010
TwoStreamNet (Ambrus et al. [ambrus2019stream])	M	640 x 192	✓	0.010 $\pm$ 0.002	0.009 $\pm$ 0.002
PackNet-SfM	M	640 x 192	✓	0.011 $\pm$ 0.006	0.009 $\pm$ 0.007
PackNet-SfM	M+v	640 x 192	✓	0.010 $\pm$ 0.005	0.009 $\pm$ 0.008
PackNet-SfM	M+v	640 x 192		0.014 $\pm$ 0.007	0.012 $\pm$ 0.008

Full text

3D Packing for Self-Supervised Monocular Depth Estimation:

Supplementary Material

Vitor Guizilini  Rare\cbs Ambru\cbs  Sudeep Pillai  Allan Raventos  Adrien Gaidon

Toyota Research Institute (TRI)

first.lastname@tri.global

1 Pose evaluation

In Table 1 we show the results of our proposed PackNet-SfM framework on the KITTI odometry benchmark [geiger2013vision]. To compare with related methods, we train our framework from scratch on sequences 00-08 of the KITTI odometry benchmark, with exactly the same parameters and networks used for depth evaluation (Table 3, main text). For consistency with related methods, we compute the Absolute Trajectory Error (ATE) averaged over all overlapping 5-frame snippets on sequences 09 and 10. Note that our pose network only takes two frames as input, and outputs a single transformation between that pair of frames. To evaluate our model on 5-frame snippets we combine the relative transformations between the target frame and the first context frame into 5-frame long overlapping trajectories, i.e. we stack $f_{x}\left(I_{t},I_{t-1}\right)=x_{t\to t-1}$ to create appropriately sized trajectories.

The ATE results are summarized in Table 1, with our proposed framework achieving competitive results relative to other related methods. We also note that all these related methods are trained in the monocular setting (M), and therefore scaled at test-time using ground truth information. Our method, on the other hand, when trained with the proposed velocity supervision loss (M+v) does not require ground-truth scaling at test-time, as it is able to recover metrically accurate scale purely from monocular imagery. Nevertheless, it is still able to achieve competitive results compared to other methods. Examples of reconstructed trajectories obtained using PackNet-SfM for the test sequences can be found in Figure 1.

2 Dense Depth for Automated Driving (DDAD)

In this section, we provide a brief overview of our newly introduced DDAD (Dense Depth for Automated Driving) dataset and the relevant properties that make it desirable as a dense depth estimation benchmark. It includes a high-resolution, long-range Luminar-H2111https://www.luminartech.com/technology as the LiDAR sensor used to generate pointclouds, with a maximum range of 250m and sub-1cm range precision. Additionally, it contains six calibrated cameras time-synchronized at 10 Hz, that together produce a 360∘ coverage around the vehicle. Note that in our work we only use information from the front-facing camera for training and evaluation.

Examples of a Luminar-H2 pointcloud projected onto each of these six cameras are shown in Figures 2,  3 and  4, for different urban settings. The depth maps generated from projecting these Luminar pointclouds onto the camera frame allow us to evaluate depth estimation methods in a much more challenging way, both in terms of denseness and longer ranges. In Table 2 and Figure 6 of the main text we show how our proposed PackNet architecture outperforms other related methods under these conditions. In fact, the gap in performance increases when considering denser ground-truth information at longer ranges, both on the entire interval and at discretized bins.

DDAD is a cross-continental dataset with scenes drawn from urban settings in the United States (San Francisco Bay Area, Detroit and Ann Arbor) and Japan (Tokyo and Odaiba). Each scene is 5 or 10 seconds long and consists of 50 or 100 samples with corresponding Luminar-H2 pointcloud and six image frames, including intrinsic and extrinsic calibration. The training set contains 194 scenes with a total of 17050 individual samples, and the validation set contains 60 senes with a total of 4150 samples. The six cameras are $2.4$ MP ($1936\times 1216$), global-shutter, and oriented at $60\degree$ intervals. They are synchronized with $10$ Hz scans from our Luminar-H2 sensors oriented at $90\degree$ intervals.