Neural RGB->D Sensing: Depth and Uncertainty from a Video Camera

TL;DR

This paper introduces a deep learning approach that transforms a monocular video into a reliable RGB-D sensing system by estimating per-pixel depth distributions and accumulating them over time to improve accuracy and stability.

Contribution

It presents a novel method for continuous depth and uncertainty estimation from monocular videos, using depth probability distributions and Bayesian filtering, outperforming prior approaches.

Findings

Achieves more accurate depth estimates than previous methods.

Provides stable and robust depth predictions over time.

Enables integration with classical 3D reconstruction techniques.

Abstract

Depth sensing is crucial for 3D reconstruction and scene understanding. Active depth sensors provide dense metric measurements, but often suffer from limitations such as restricted operating ranges, low spatial resolution, sensor interference, and high power consumption. In this paper, we propose a deep learning (DL) method to estimate per-pixel depth and its uncertainty continuously from a monocular video stream, with the goal of effectively turning an RGB camera into an RGB-D camera. Unlike prior DL-based methods, we estimate a depth probability distribution for each pixel rather than a single depth value, leading to an estimate of a 3D depth probability volume for each input frame. These depth probability volumes are accumulated over time under a Bayesian filtering framework as more incoming frames are processed sequentially, which effectively reduces depth uncertainty and improves accuracy, robustness, and temporal stability. Compared to prior work, the proposed approach achieves more accurate and stable results, and generalizes better to new datasets. Experimental results also show the output of our approach can be directly fed into classical RGB-D based 3D scanning methods for 3D scene reconstruction.

Figures

Tables

Table 1. Table 1: Comparison of depth estimation over the 7-Scenes dataset  [ 43 ] with the metrics defined in  [ 11 ] .

		abs. rel	rmse	scale inv.
DeMoN [51]	31.88	0.3888	0.8549	0.4473
DORN [13]	60.05	0.2000	0.4591	0.2207
Ours	69.26	0.1758	0.4408	0.1899

Table 2. Table 2: Comparison of depth estimation on KITTI  [ 16 ] .

		abs. rel	rmse	scale inv.
Eigen [11]	67.80	0.1904	5.114	0.2628
Mono [17]	86.43	0.1238	2.8684	0.1635
DORN [13]	92.62	0.0874	3.1375	0.1233
Ours	93.15	0.0998	2.8294	0.1070

Table 3. Table 3: Cross-dataset tests for depth estimation in the outdoors.

	KITTI (train) $\to$ virtual KITTI (test)
		abs. rel	rmse	scale inv.
DORN [13]	69.61	0.2256	9.618	0.3986
Ours	73.38	0.2537	6.452	0.2548
	Indoor (train) $\to$ KITTI (test)
		abs. rel	rmse	scale inv.
DORN [13]	25.44	0.6352	8.603	0.4448
Ours	72.96	0.2798	5.437	0.2139

Table 4. Table 4: Performance on 7Scenes with different initial poses

		abs. rel	rmse	scale inv.
VO pose	60.63	0.1999	0.4816	0.2158
$1$st win.	62.08	0.1923	0.4591	0.2001
GT $R$	69.26	0.1758	0.4408	0.1899
GT pose	70.54	0.1619	0.3932	0.1586

Table 5. Table 5: Comparison of depth estimation over the 7-Scenes dataset  [ 43 ] with the metrics defined in  [ 11 ]

				abs. rel	sq. rel	rmse	rmse log	scale. inv
DeMoN [51]	31.88	61.02	82.52	0.3888	0.4198	0.8549	0.4771	0.4473
DORN [13]	60.05	87.76	96.33	0.2000	0.1153	0.4591	0.2813	0.2207
Ours	69.26	91.77	96.82	0.1758	0.1123	0.4408	0.2500	0.1899

Table 6. Table 6: Comparison of depth estimation over the KITTI dataset  [ 16 ] .

				abs. rel	sq. rel	rmse	rmse log	scale. inv
Eigen [11]	67.80	88.79	96.51	0.1904	1.263	5.114	0.2758	0.2628
Mono [17]	86.43	97.70	99.47	0.1238	0.5023	2.8684	0.1644	0.1635
DORN [13]	92.62	98.18	99.35	0.0874	0.4134	3.1375	0.1337	0.1233
Ours	93.15	98.018	99.25	0.0998	0.4732	2.8294	0.1280	0.1070

Table 7. Table 7: Performance on KITTI dataset without GPS/IMU measurements

				abs. rel	sq. rel	rmse	rmse log	scale. inv
GT R	89.34	98.30	99.64	0.1178	0.4490	3.2042	0.1514	0.1509
opt. pose	87.78	97.22	99.10	0.1201	0.5763	3.5157	0.1672	0.1665

Table 8. Table 8: K-Net structure. The operator expand( ⋅ ⋅ \cdot ) repeat the image intensity in the depth dimension

Name	Components	Input	Output dimension
Input	concat(cost_volume, expand($I_{\text{ref}}$))		$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ D $\times$ 4
conv_0	conv_3d(3$\times$3, ch_in=4, ch_out=32), ReLU conv_3d(3$\times$3, ch_in=32, ch_out=32), ReLU	Input	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ D $\times$ 32
conv_1	$\left[\begin{array}{c}\text{conv\_3d(3 }\times \text{3, ch\_in=32, ch\_out=32), ReLU}\\ \text{conv\_3d(3}\times \text{3, ch\_in=32, ch\_out=32) }\end{array}\right]$ $\times$ 4	conv_0	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ D $\times$ 32
conv_2	conv_3d(3$\times$3, ch_in=32, ch_out=32), ReLU conv_3d(3$\times$3, ch_in=32, ch_out=1)	conv_1	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ D $\times$ 1
Output	Modified cost_volume from the conv_2 layer		$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ D $\times$ 1

Table 9. Table 9: R-Net structure

Name	Components	Input	Output dimension
Input	cost_volume from K-Net		$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ D
conv_0	conv_2d(3$\times$3, ch_in=64$+$D, ch_out= 64$+$D), LeakyReLU conv_2d(3$\times$3, ch_in=64$+$D, ch_out= 64$+$D), LeakyReLU	concat(Input, fusion in D-Net )	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ (64$+$D)
trans_conv_0	transpose_conv(4$\times$4, ch_in=64$+$D, ch_out=D, stride=2), LeakyReLU	conv_0	$\frac{1}{2}$H $\times$ $\frac{1}{2}$ W $\times$ D
conv_1	conv_2d(3$\times$3, ch_in=32$+$D, ch_out=32 $+$ D ), LeakyReLU conv_2d(3$\times$3, ch_in=32$+$D, ch_out=32 $+$ D),LeakyReLU	concat(trans_conv_0, conv_1 in D-Net	$\frac{1}{2}$H $\times$ $\frac{1}{2}$ W $\times$ (D$+$32)
trans_conv_1	transpose_conv(4$\times$4, ch_in=32$+$D, ch_out=D, stride=2 ), LeakyReLU	conv_1	H $\times$ W $\times$ D
conv_2	conv_2d(3$\times$3, ch_in=3$+$D, ch_out=3$+$D ), LeakyReLU conv_2d(3$\times$3, ch_in=3$+$D, ch_out=D ), LeakyReLU conv_2d(3$\times$3, ch_in= D, ch_out=D )	concat(trans_conv_1, $I_{\text{ref}}$)	H $\times$ W $\times$ D
Output	Upsampled and refined cost_volume		H $\times$ W $\times$ D

Table 10. Table 10: D-Net structure. The structure is taken from [ 6 ]

Name	Components	Input	Output dimension
Input	Input frame		H $\times$ W $\times$ 3
CNN Layers
conv0_1	conv_2d(3$\times$3, ch_in=3, ch_out=32, stride=2), ReLU	Input	$\frac{1}{2}$H $\times$ $\frac{1}{2}$ W $\times$ 32
conv0_2	conv_2d(3$\times$3, ch_in=32, ch_out=32 ), ReLU	conv0_1	$\frac{1}{2}$H $\times$ $\frac{1}{2}$ W $\times$ 32
conv0_3	conv_2d(3$\times$3, ch_in=32, ch_out=32), ReLU	conv0_2	$\frac{1}{2}$H $\times$ $\frac{1}{2}$ W $\times$ 32
conv1	$\left[\begin{array}{c}\text{conv\_2d(3}\times \text{3, ch\_in=32, ch\_out=32), ReLU}\\ \text{conv\_2d(3}\times \text{3, ch\_in=32, ch\_out=32)}\end{array}\right]$ $\times$ 3	conv0_2	$\frac{1}{2}$H $\times$ $\frac{1}{2}$ W $\times$32
conv1_1	conv_2d(3$\times$3, ch_in=32, ch_out=64, stride=2), ReLU	conv1	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$64
conv2	$\left[\begin{array}{c}\text{conv\_2d(3}\times \text{3, ch\_in=64, ch\_out=64), ReLU}\\ \text{conv\_2d(3}\times \text{3, ch\_in=64, ch\_out=64)}\end{array}\right]$ $\times$ 15	conv1_1	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$64
conv2_1	conv_2d(3$\times$3, ch_in=64, ch_out=128), ReLU	conv2	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$128
conv3	$\left[\begin{array}{c}\text{conv\_2d(3}\times \text{3, ch\_in=128, ch\_out=128), ReLU}\\ \text{conv\_2d(3}\times \text{3, ch\_in=128, ch\_out=128)}\end{array}\right]$ $\times$ 2	conv2_1	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 128
conv4	$\left[\begin{array}{c}\text{conv\_2d(3}\times \text{3, ch\_in=128, ch\_out=128, dila=2), ReLU}\\ \text{conv\_2d(3}\times \text{3, ch\_in=128, ch\_out=128, dila=2)}\end{array}\right]$ $\times$ 3	conv3	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 128
Spatial Pyramid Layers
branch1	avg_pool(64$\times$64,stride=64) conv_2d(1$\times$1, ch_in=128, ch_out=32), ReLU bilinear interpolation	conv4	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 32
branch2	avg_pool(32 $\times$ 32,stride= 32) conv_2d(1$\times$1, ch_in=128, ch_out=32), ReLU bilinear interpolation	conv4	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 32
branch3	avg_pool(16 $\times$ 16,stride= 16) conv_2d(1$\times$1, ch_in=128, ch_out=32), ReLU bilinear interpolation	conv4	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 32
branch4	avg_pool(8 $\times$ 8,stride= 8) conv_2d(1$\times$1, ch_in=128, ch_out=32), ReLU bilinear interpolation	conv4	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 32
concat	concat(branch1, branch2, branch3, branch4, conv2, conv4)		$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 320
fusion	conv_2d(3$\times$3, ch_in=320, ch_out=128), ReLU conv_2d(1$\times$1, ch_in=128, ch_out=64), ReLU	concat	$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 64
Output	The extracted image feature from the fusion layer		$\frac{1}{4}$H $\times$ $\frac{1}{4}$ W $\times$ 64

Equations

$$\text{mbox}Depth:$$

$$\text{mbox}Confidence:$$

$$L(d_t|I_t)$$

$$p(d_t|I_t)$$

$$\text{mbox}Predict:$$

$$\text{mbox}Update:$$

$$p(d_{t+1}|I_{1:t})=\text{mbox}warp(p(d_t|I_{1:t}),\delta T_{t,t+1}),$$

$$p(d_{t+1}|I_{1:t+1})=p(d_{t+1}|I_{1:t})\cdot p(d_{t+1}|I_{t+1}).$$

$$E(d_{t+1}|I_{1:t+1})=E(d_{t+1}|I_{1:t})+E(d_{t+1}|I_{t+1}),$$

$$E(d_{t+1}|I_{1:t+1})=\lambda \cdot E(d_{t+1}|I_{1:t})+E(d_{t+1}|I_{t+1}).$$

$$E(d_{t+1}|I_{1:t+1})=E(d_{t+1}|I_{1:t})+g(\Delta E_{t+1},I_{t+1}),$$

$$\Delta E_{t+1}=E(d_{t+1}|I_{t+1})-E(d_{t+1}|I_{1:t}),$$

$$\text{vspace}1em\underset{\begin{array}{c}\delta T_{k,t+1}\\ k\in {\mathcal{N}}_{t+1},k\ne t+1\end{array}}{\text{min.}}\text{vspace}1em\sum _k\hat{C}|I_{t+1}-\text{warp}(I_k;\hat{d};\delta T_{k,t+1}){|}_1,$$

$$h_t=W_t{h}_{t-1}+K_t(x_t-V_t{W}_t{h}_{t-1})$$

$$h_t$$

$$=W_t{h}_{t-1}+KV(h_t-W_t{h}_{t-1})$$

Full text

1 Introduction

Depth sensing is crucial for 3D reconstruction [32, 33, 53] and scene understanding [18, 35, 44]. Active depth sensors (e.g., time of flight cameras [19, 36], LiDAR [7]) measure dense metric depth, but often have limited operating range (e.g., indoor) and spatial resolution [5], consume more power, and suffer from multi-path reflection and interference between sensors [29]. In contrast, estimating depth directly from image(s) solves these issues, but faces other long-standing challenges such as scale ambiguity and drift for monocular methods [38], as well as the correspondence problem and high computational cost for stereo [48] and multi-view methods [42].

Inspired by recent success of deep learning in 3D vision [4, 6, 13, 17, 20, 47, 51, 52, 54, 56, 57], in this paper, we propose a DL-based method to estimate depth and its uncertainty continuously from a monocular video stream, with the goal of effectively turning an RGB camera into an RGB-D camera. We have two key ideas:

Unlike prior work, for each pixel, we estimate a depth probability distribution rather than a single depth value, leading to an estimate of a Depth Probability Volume (DPV) for each input frame. As shown in Fig. 1, the DPV provides both a Maximum-Likelihood-Estimate (MLE) of the depth map, as well as the corresponding per-pixel uncertainty measure. 2. 2.

These DPVs across different frames are accumulated over time, as more incoming frames are processed sequentially. The accumulation step, originated from the Bayesian filtering theory and implemented as a learnable deep network, effectively reduces depth uncertainty and improves accuracy, robustness, and temporal stability over time, as shown later in Sec. 4.

We argue that all DL-based depth estimation methods should predict not depth values but depth distributions, and should integrate such statistical distributions over time (e.g., via Bayesian filtering). This is because dense depth estimation from image(s) – especially for single-view methods – inherently has a lot of uncertainty, due to factors such as lack of texture, specular/transparent material, occlusion, and scale drift. While some recent work started focusing on uncertainty estimation [15, 21, 23, 24] for certain computer vision tasks, to our knowledge, we are the first to predict a depth probability volume from images and integrate it over time in a statistical framework.

We evaluate our method extensively on multiple datasets and compare with recent state-of-the-art, DL-based, depth estimation methods [13, 17, 51]. We also perform the so-called “cross-dataset” evaluation task, which tests models trained on a different dataset without fine-tuning. We believe such cross-dataset tasks are essential to evaluate the robustness and generalization ability [1]. Experimental results show that, with reasonably good camera pose estimation, our method outperforms these prior methods on depth estimation with better accuracy, robustness, and temporal stability. Moreover, as shown in Fig. 1, the output of the proposed method can be directly fed into RGB-D based 3D scanning methods [32, 33] for 3D scene reconstruction.

2 Related Work

Depth sensing from active sensors

Active depth sensors, such as depth cameras [19, 36] or LiDAR sensors [7] provide dense metric depth measurements as well as sensor-specific confidence measure [37]. Despite of their wide usage [18, 32, 35, 53], they have several inherent drawbacks[5, 29, 34, 50], such as limited operating range, low spatial resolution, sensor interference, and high power consumption. Our goal in this paper is to mimic a RGB-D sensor with a monocular RGB camera, which continuously predicts depth (and its uncertainty) from a video stream.

Depth estimation from images

Depth estimation directly from images has been a core problem in computer vision [39, 42]. Classical single view methods [9, 38] often make strong assumptions on scene structures. Stereo and multi-view methods [42] rely on triangulation and suffer from finding correspondences for textureless regions, transparent/specular materials, and occlusion. Moreover, due to global bundle adjustment, these methods are often computationally expensive for real-time applications. For depth estimation from a monocular video, there is also scale ambiguity and drifting [30]. Because of these challenges, many computer vision systems [30, 40] use RGB images mainly for camera pose estimation but rarely for dense 3D reconstruction [41]. Nevertheless, depth sensing from images has great potentials, since it addresses all the above drawbacks of active depth sensors. In this paper, we take a step in this direction using a learning-based method.

Learning-based depth estimation

Recently researchers have shown encouraging results for depth sensing directly from images(s), including single-view methods [13, 17, 57], video-based methods [28, 52, 55], depth and motion from two views [6, 51], and multi-view stereo [20, 54, 56]. A few work also incorporated these DL-based depth sensing methods into visual SLAM systems [4, 47]. Despite of the promising performance, however, these DL-based methods are still far from real-world applications, since their robustness and generalization ability is yet to be thoroughly tested [1]. In fact, as shown in Sec. 4, we found many state-of-the-art methods degrade significantly even for simple cross-dataset tasks. This gives rise to an increasing demand for a systematic study of uncertainty and Bayesian deep learning for depth sensing, as performed in our paper.

Uncertainty and Bayesian deep learning

Uncertainty and Bayesian modeling have been long studied in last few decades, with various definitions ranging from the variance of posterior distributions for low-level vision [46] and motion analysis [25] to variability of sensor input models [22]. Recently, uncertainty [15, 23] for Bayesian deep learning were introduced for a variety of computer vision tasks [8, 21, 24]. In our work, the uncertainty is defined as the posterior probability of depth, i.e., the DPV estimated from a local window of several consecutive frames. Thus, our network estimates the “measurement uncertainty” [23] rather than the “model uncertainty”. We also learn an additional network module to integrate this depth probability distribution over time in a Bayesian filtering manner, in order to improve the accuracy and robustness for depth estimation from a video stream.

3 Our Approach

Figure 2 shows an overview of our proposed method for depth sensing from an input video stream, which consists of three parts. The first part (Sec. 3.1) is the D-Net, which estimates the Depth Probability Volume (DPV) for each input frame. The second part (Sec. 3.2) is the K-Net, which helps to integrate the DPVs over time. The third part (Sec. 3.3) is the refinement R-Net, which improves the spatial resolution of DPVs with the guidance from input images.

Specifically, we denote the depth probability volume (DPV) as $p(d;u,v)$, which represents the probability of pixel $(u,v)$ having a depth value $d$, where $d\in[d_{min},d_{max}]$. Due to perspective projection, the DPV is defined on the 3D view frustum attached to the camera, as shown in Fig. 3(a). $d_{min}$ and $d_{max}$ are the near and far planes of the 3D frustum, which is discretized into $N=64$ planes uniformly over the inverse of depth (i.e., disparity). The DPV contains the complete statistical distribution of depth for a given scene. In this paper, we directly use the non-parametric volume to represent DPV. Parametric models, such as Gaussian Mixture Model [3], can be also be used. Given the DPV, we can compute the Maximum-Likelihood Estimates (MLE) for depth and its confidence:

[TABLE]

To make notations more concise, we will omit $(u,v)$ and use $p(d)$ for DPVs in the rest of the paper.

When processing a video stream, the DPV can be treated as a hidden state of the system. As the camera moves, as shown in Fig. 3(b), the DPV $p(d)$ is being updated as new observations arrive, especially for the overlapping volumes. Meanwhile, if camera motion is known, we can easily predict the next state $p(d)$ from the current state. This predict-update iteration naturally implies a Bayesian filtering scheme to update the DPV over time for better accuracy.

3.1 D-Net: Estimating DPV

For each frame $I_{t}$, we use a CNN, named D-Net, to estimate the conditional DPV, $p(d_{t}|I_{t})$, using $I_{t}$ and its temporally neighboring frames. In this paper, we consider a local time window of five frames $\mathcal{N}_{t}=[t-2\Delta t,t-\Delta t,t,t+\Delta t,t+2\Delta t]$, and we set $\Delta t=5$ for all our testing videos (25fps/30fps). For a given depth candidate $d$, we can compute a cost map by warping all the neighboring frames into the current frame $I_{t}$ and computing their differences. Thus, for all depth candidates, we can compute a cost volume, which produces the DPV after a softmax layer:

[TABLE]

where $f(\cdot)$ is a feature extractor, $\delta T_{kt}$ is the relative camera pose from frame $I_{k}$ to frame $I_{t}$, $\mbox{warp}(\cdot)$ is an operator that warps the image features from frame $I_{k}$ to the reference frame $I_{t}$, which is implemented as 2D grid sampling. In this paper, without loss of generality, we use the feature extractor $f(\cdot)$ from PSM-Net [6], which outputs a feature map of 1/4 size of the input image. Later in Sec. 3.3, we learn a refinement R-Net to upsample the DPV back to the original size of the input image.

Figure 4 shows an example of a depth map $\hat{d}(u,v)$ and its confidence map $\hat{C}(u,v)$ (blue means low confidence) derived from a Depth Probability Volume (DPV) from the input image. The bottom plot shows the depth probability distributions $p(d;u,v)$ for the three selected points, respectively. The red and green points have sharp peaks, which indicates high confidence in their depth values. The blue point is in the highlight region, and thus it has a flat depth probability distribution and a low confidence for its depth.

3.2 K-Net: Integrating DPV over Time

When processing a video stream, our goal is to integrate the local estimation of DPVs over time to reduce uncertainty. As mentioned earlier, this integration can be naturally implemented as Bayesian filtering. Let us define $d_{t}$ as the hidden state, which is the depth (in camera coordinates) at frame $I_{t}$. The “belief” volume $p(d_{t}|I_{1:t})$ is the conditional distribution of the state giving all the previous frames. A simple Bayesian filtering can be implemented in two iterative steps:

[TABLE]

where the prediction step is to warp the current DPV from the camera coordinate at $t$ to the camera coordinate at $t+1$:

[TABLE]

where $\delta T_{t,t+1}$ is the relative camera pose from time $t$ to time $t+1$, and $\mbox{warp}(\cdot)$ here is a warping operator implemented as 3D grid sampling. At time $t+1$, we can compute the local DPV $p(d_{t+1}|I_{t+1})$ from the new measurement $I_{t+1}$ using the D-Net. This local estimate is thus used to update the hidden state, i.e., the “belief” volume,

[TABLE]

Note that we always normalize the DPV in the above equations and ensure $\int_{d_{min}}^{d_{max}}p(d)=1$. Figure 5 shows an example. As shown in the second row, with the above Bayesian filtering (labeled as ”no damping”), the estimated depth map is less noisy, especially in the regions of the back wall and the floor.

However, one problem with directly applying Bayesian filtering is it integrates both correct and incorrect information in the prediction step. For example, when there are occlusions or disocclusions, the depth values near the occlusion boundaries change abruptly. Applying Bayesian filtering directly will propagate wrong information to the next frames for those regions, as highlighted in the red box in Fig. 5. One straightforward solution is to reduce the weight of the prediction in order to prevent incorrect information being integrated over time. Specifically, by defining $E(d)=-\log p(d)$, Eq. 6 can be re-written as

[TABLE]

where the first term is the prediction and the second term is the measurement. To reduce the weight of the prediction, we multiply a weight $\lambda\in[0,1]$ with the first term,

[TABLE]

We call this scheme “global damping”. As shown in Fig. 5, global damping helps to reduce the error in the disoccluded regions. However, global damping may also prevent some correct depth information to be integrated to the next frames, since it reduces the weights equally for all voxels in the DPV. Therefore, we propose an “adaptive damping” scheme to update the DPV:

[TABLE]

where $\Delta E_{t+1}$ is the difference between the measurement and the prediction,

[TABLE]

and $g(\cdot)$ is a CNN, named K-Net, which learns to transform $\Delta E_{t+1}$ into a correction term to the prediction. Intuitively, for regions with correct depth probability estimates, the values in the overlapping volume of DPVs are consistent. Thus the residual in Eq. 9 is small and the DPV will not be updated in Eq. 8. On the other hand, for regions with incorrect depth probability, the residual would be large and the DPV will be corrected by $g(\Delta E,I_{t+1})$. This way, the weight for prediction will be changed adaptively for different DPV voxels. As shown in Fig. 5, the adaptive damping, i.e., K-Net, significantly improve the accuracy for depth estimation. In fact, K-Net is closely related to the derivation of Kalman filter, where “K” stands for Kalman gain. Please refer to the appendix for details.

3.3 R-Net and Training Details

Finally, since the DPV $p(d_{t}|I_{1:t})$ is estimated with $1/4$ spatial resolution (on both width and height) of the input image, we employ a CNN, named R-Net, to upsample and refine the DPV back to the original image resolution. The R-Net, $h(\cdot)$, is essentially an U-Net with skip connections, which takes input the low-res DPV from the K-Net $g(\cdot)$ and the image features extracted from the feature extractor $f(\cdot)$, and outputs a high-resolution DPV.

In summary, as shown in Fig. 2, the entire network has three modules, i.e., the D-Net, $f(\cdot;\Theta_{1})$, the K-Net, $g(\cdot;\Theta_{2})$, and the R-Net, $h(\cdot;\Theta_{3})$. Detailed network architectures are provided in the appendix. The full network is trained end-to-end, with simply the Negative Log-Likelihood (NLL) loss over the depth, $\mbox{Loss}=\mbox{NLL}(p(d),d_{GT})$. We also tried to add image warping as an additional loss term (i.e., minimizing the difference between $I_{t}$ and the warped neighboring frames), but we found that it does not improve the quality of depth prediction.

During training, we use ground truth camera poses. For all our experiments, we use the ADAM optimizer [26] with a learning rate of $10^{-5}$, $\beta_{1}=.9$ and $\beta_{2}=.999$. The whole framework, including D-Net, K-Net and R-Net, is trained together in an end-to-end fashion for 20 epochs.

3.4 Camera Poses during Inference

During inference, given an input video stream, our method requires relative camera poses $\delta T$ between consecutive frames — at least for all the first five frames — to bootstrap the computation of the DPV. In this paper, we evaluated several options to solve this problem. In many applications, such as autonomous driving and AR, initial camera poses may be provided by additional sensors such as GPS, odometer, or IMU. Alternatively, we can also run state-of-the-art monocular visual odometry methods, such as DSO [12], to obtain the initial camera poses. Since our method outputs continuous dense depth maps and their uncertainty maps, we can in fact further optimize the initial camera poses within a local time window, similar to local bundle adjustment [49].

Specifically, as shown in Fig. 6, given $p(d_{t}|I_{1:t})$, the DPV of the reference frame $I_{t}$ in the local time window $\mathcal{N}_{t}$, we can warp $p(d_{t}|I_{1:t})$ to the reference camera view in $\mathcal{N}_{t+1}$ to predict the DPV $p(d_{t+1}|I_{1:t})$ using Eq. 5. Then we get the depth map $\hat{d}$ and confidence map $\hat{C}$ for the new reference frame using Eq. 2. The camera poses within the local time window $\mathcal{N}_{t+1}$ are optimized as:

[TABLE]

where $\delta T_{k,t+1}$ is the relative camera pose of frame $k$ to frame $t+1$; $I_{k}$ is the source image at frame $k$; $\mbox{warp}(\cdot)$ is an operator that warps the image from the source view to the reference view.

4 Experimental Results

We evaluate our method on multiple indoor and outdoor datasets [43, 45, 14, 16], with an emphasis on accuracy and robustness. For accuracy evaluation, we argue the widely-used statistical metrics [11, 51] are insufficient because they can only provide an overall estimate over the entire depth map. Rather, we feed the estimated depth maps directly into classical RGB-D based 3D scanning systems [32, 33] for 3D reconstruction — this will show the metric accuracy, the consistency, and the usefulness of the estimation. For robustness evaluation, we performed the aforementioned cross-dataset evaluation tasks, i.e., testing on new datasets without fine-tuning. The performance degradation over new datasets will show the generalization ability and robustness for a given algorithm.

As no prior work operates in the exact setting as ours, it is difficult to choose methods to compare with. We carefully select a few recent DL-based depth estimation methods and try our best for a fair comparison. For single-view methods, we select DORN [13] which is the current state-of-the-art [1]. For two-view methods, we compare with DeMoN [51], which shows high quality depth prediction from a pair of images. We also compare with MonoDepth [17], which is a semi-supervised learning approach from stereo images. To improve the temporal consistency for these per-frame estimations, we trained a post-processing network [27], but we observed it does not improve the performance. Since there is always scale ambiguity for depth from a monocular camera, for fair comparison, we normalize the scale for the outputs from all the above methods before we compute statistical metrics [11].

The inference time for processing one frame in our method is $\sim$ $0.7$ second per frame without pose optimization and $\sim$ $1.5$ second with pose estimation on a workstation with GTX 1080 GPU and 64 GB RAM memory, with the framework implemented in Python. The pose estimation part can be implemented with C++ to improve efficiency.

Results for Indoor Scenarios

We first evaluated our method for indoor scenarios, for which RGB-D sensors were used to capture dense metric depth for ground truth. We trained our network on ScanNet [10]. Figure 7 shows two exemplar results. As shown, in addition to depth maps, our method also outputs reasonable confidence maps (e.g., low confidence in the occluded or specular regions) which correlates with the depth errors. Moreover, with more input frames, the confidence maps accumulate correctly over time: the confidence of the books (top row) increases and the depth error decreases; the confidence of the glass region (bottom row) decreases and the depth error increases.

For comparison, since the models provided by DORN and DeMoN were trained on different datasets, we compare with these two methods on a separate indoor dataset 7Scenes [43]. For our method, we assume that the relative camera rotation $\delta R$ within a local time window is provided (e.g. measured by IMU). As shown in Table 5, our method significantly outperforms both DeMoN and DORN on this dataset based on the commonly used statistical metrics [11]. We include the complete metrics in the appendix. Without using an IMU, our method can also achieve better performance, as shown in Table 4.

For qualitative comparison, as shown in Fig. 8, the depth maps from our method are less noisy, more sharper, and temporally more consistent. More importantly, using an RGB-D 3D scanning method [33], we can reconstruct a much higher quality 3D mesh with our estimated depths compared to other methods. Even when compared with 3D reconstruction using a real RGB-D sensor, our result has better coverage and accuracy in some regions (e.g., monitors / glossy surfaces) where active depth sensors cannot capture.

Results for Outdoor Scenarios

We also evaluated our method on some outdoor datasets — KITTI [16] and virtual KITTI [14]. The virtual KITTI dataset is used because it has dense, accurate metric depth as ground truth, while KITTI only has sparse depth values from LiDAR as ground truth. For our method, we use the camera poses measured by the IMU and GPS. Table 6 lists the comparison results with DORN [13], Eigen [11], and MonoDepth [17] which are also trained on KITTI [16]. Our method has similar performance with DORN [13], and is better than the other two methods, based on the statistical metrics defined in [11]. We also tested our method with camera poses from DSO [12] and obtain slightly worse performance (see appendix).

Figure 9 shows qualitative comparison for depth maps in KITTI dataset. As shown, our method generate sharper and less noisier depth maps. In addition, our method outputs depth confidence maps (e.g., lower confidence on the car window). Our depth estimation is temporally consistent, which leads to the possibility of fusing multiple depth maps with voxel hashing [33] in the outdoors for a large-scale dense 3D reconstruction, as shown in Fig. 9.

In Table 3, we performed the cross-dataset task. The left shows the results with training from KITTI [16] and testing on virtual KITTI [14]. The right shows the results with training from indoor datasets (NYUv2 [31] for DORN [13] and ScanNet [10] for ours) and testing on KITTI [16]. As shown, our method performs better, which shows its better robustness and generalization ability.

Ablation Study

The performance of our method relies on accurate estimate of camera poses, so we test our method with different camera pose estimation schemes: (1) Relative camera rotation $\delta R$ is read from an IMU sensor (denoted as “GT $R$”). (2) $\delta R$ of all frames are initialized with DSO [12] (denoted as “VO pose”) (3) $\delta R$ of the first five frames are initialized with DSO [12] (denoted as “$1$st win”). We observe that when only the camera poses in the first time window are initialized using DSO, the performance in terms of depth estimation is better than that using the DSO pose initialization for all frames. This may seem counter-intuitive, but it is because monocular VO methods sometimes have large errors for textureless regions while optimization with dense depths may overcome this problem.

Usefulness of the Confidence Map

The estimated confidence maps can also be used to further improve the depth maps. As shown in Fig. 10(a), given the depth map and the corresponding confidence, we can correct the regions with lower confidence due to specular reflection. Also, for 3D reconstruction algorithm, given the depth confidence, we can mask out the regions with lower confidence for better reconstruction, as shown in Fig. 10(b).

5 Conclusions and Limitations

In this paper, we present a DL-based method for continuous depth sensing from a monocular video camera. Our method estimates a depth probability distribution volume from a local time window, and integrates it over time under a Bayesian filtering framework. Experimental results show our approach achieves high accuracy, temporal consistency, and robustness for depth sensing, especially for the cross-dataset tasks. The estimated depth maps from our method can be fed directly into RGB-D scanning systems for 3D reconstruction and achieve on-par or sometimes more complete 3D meshes than using a real RGB-D sensor.

There are several limitations that we plan to address in the future. First, camera poses from a monocular video often suffer from scale drifting, which may affect the accuracy of our depth estimation. Second, in this work we focus on depth sensing from a local time window, rather than solving it in a global context using all the frames.

Appendix A Relation of K-Net to the Kalman filter

The proposed update process defined in Eq. 8 in the main paper using residuals is closely related to Kalman Filter. In Kalman Filter, given the observation $x_{t}$ at time $t$ and the estimated hidden state $h_{t-1}$ at time $t-1$, the updated hidden state $h_{t}$ is:

[TABLE]

where $W_{t}$ is the transition matrix mapping the previous hidden state to current state; $K_{t}$ is the gain matrix mapping the residual in the observation space to the hidden state space. $V_{t}$ is the measurement matrix mapping the estimation in the hidden state space back to the observation space.

If we assume the measurement matrix is accurate: $x_{t}=Vh_{t}$, and the gain and measurement matrices are temporally invariant, we have:

[TABLE]

Comparing our proposed update process in Eq. 5, Eq. 8 and Eq. 9 in the main paper and Kalman Filter in Eq.12, in our case the input images correspond to the observations $x_{t}$ ; the negative-log depth probabilities correspond to the hidden states $h_{t}$; the warping operator $\text{warp}(\cdot)$ corresponds to the transition matrix $W_{t}$; the K-Net $g(\cdot)$ corresponds to the multiplication of the gain and measurement matrices $KV$ in Eq. 12.

Appendix B More Results

B.1 Complete metrics for Comparisons

We show the complete metrics for depth estimation comparisons in Table 5 and Table 6.

B.2 Results on KITTI without GPS or IMU

In Table 7, we show the performance of our method on the KITTI dataset, in case where only the IMU measurement are available (denoted as ’GT R’), and neither IMU nor GPU are available (denoted as ’opt. pose’).

Appendix C Network structures

In this section, we illustrate the network structures used in the pipeline.

C.1 D-Net

We show the structure of the D-Net in Table. 10. In the paper, we set $D=64$.

C.2 K-Net

We show the structure of the K-Net in Table. 8. In the paper, we set $D=64$.

C.3 R-Net

We show the structure of the R-Net in Table. 9. In the paper, we set $D=64$.