FastDepth: Fast Monocular Depth Estimation on Embedded Systems

TL;DR

FastDepth is a lightweight neural network designed for real-time monocular depth estimation on embedded systems, achieving high accuracy with significantly reduced latency suitable for micro aerial vehicles.

Contribution

The paper introduces a novel efficient encoder-decoder architecture with network pruning, enabling real-time depth estimation on embedded platforms with minimal power consumption.

Findings

FastDepth runs at 178 fps on NVIDIA Jetson TX2 GPU.

Achieves near state-of-the-art accuracy on NYU Depth v2 dataset.

Operates at 27 fps on CPU with under 10 W power consumption.

Abstract

Depth sensing is a critical function for robotic tasks such as localization, mapping and obstacle detection. There has been a significant and growing interest in depth estimation from a single RGB image, due to the relatively low cost and size of monocular cameras. However, state-of-the-art single-view depth estimation algorithms are based on fairly complex deep neural networks that are too slow for real-time inference on an embedded platform, for instance, mounted on a micro aerial vehicle. In this paper, we address the problem of fast depth estimation on embedded systems. We propose an efficient and lightweight encoder-decoder network architecture and apply network pruning to further reduce computational complexity and latency. In particular, we focus on the design of a low-latency decoder. Our methodology demonstrates that it is possible to achieve similar accuracy as prior work on depth estimation, but at inference speeds that are an order of magnitude faster. Our proposed network, FastDepth, runs at 178 fps on an NVIDIA Jetson TX2 GPU and at 27 fps when using only the TX2 CPU, with active power consumption under 10 W. FastDepth achieves close to state-of-the-art accuracy on the NYU Depth v2 dataset. To the best of the authors' knowledge, this paper demonstrates real-time monocular depth estimation using a deep neural network with the lowest latency and highest throughput on an embedded platform that can be carried by a micro aerial vehicle.

Tables

Table 1. Figure 4 : Visualized results of depth estimation on the NYU Depth v2 dataset. (a) input RGB image; (b) ground truth; (c) our model, without skip connections, unpruned; (d) our model, with skip connections, unpruned; (e) our model, with skip connections, pruned; (f) error map between the output of our final pruned model and ground truth, where redder regions indicate higher error.

(a)	(b)	(c)	(d)	(e)	(f)

Table 2. TABLE VI : Hardware-specific compilation enables fast depthwise layers in our network. Runtimes are measured on the TX2.

MobileNet-NNConv5	in PyTorch		using TVM
	CPU [ms]	GPU [ms]	CPU [ms]	GPU [ms]
with standard decoder	4100	34.9	176	20.9
with depthwise decoder	5200	18.6	50	8.3
with depthwise & skip-add	5100	19.1	66	8.2

Full text

I Introduction

Depth sensing is essential to many robotic tasks, including mapping, localization, and obstacle avoidance. Existing depth sensors (e.g., LiDARs, structured-light sensors, etc.) are typically bulky, heavy, and have high power consumption. These limitations make them unsuitable for small robotic platforms (e.g., micro aerial and mini ground vehicles), which motivates depth estimation using a monocular camera, due to its low cost, compact size, and high energy efficiency.

Past research on monocular depth estimation has focused almost exclusively on improving accuracy, resulting in computation-intensive algorithms that cannot be readily adopted in robotic systems. Since most systems are not only limited in compute resources, but are also subject to latency constraints, a key challenge is balancing the computation and runtime cost with the accuracy of the algorithm.

Current state-of-the-art depth estimation algorithms rely on deep learning based methods, and while these achieve significant improvement in accuracy, they do so at the cost of increased computational complexity. Prior research on designing fast and efficient networks has primarily focused on encoder networks for tasks such as image classification and object detection [1]. In these applications, the input is an image (pixel-based), and the output is reduced to a label (an object class and position). To the best of our knowledge, little effort has been put into the efficient design of both encoder and decoder networks (i.e., auto-encoder networks) for tasks such as depth estimation, where the output is a dense image. In particular, reducing decoder complexity poses an additional challenge since there is less information reduction at each of the decoding layers and the decoder’s output is high dimensional.

To address these challenges, this paper presents a low latency, high-throughput, high-accuracy depth estimation algorithm running on embedded systems. We propose an efficient encoder-decoder network architecture with a focus on low latency design. Our approach employs MobileNet [2] as an encoder and nearest neighbor interpolation with depthwise separable convolution in the decoder. We apply state-of-the-art network pruning, NetAdapt [3], and use the TVM compiler stack [4] to further reduce inference runtime on a target embedded platform. We show that our low latency network design, FastDepth, can perform real-time depth estimation on the NVIDIA Jetson TX2 [5], operating at over 120 frames per second (fps) on the TX2 GPU (see Figure 1) and at over 25 fps on the TX2 CPU only111This throughput is achieved with a batch size of one and 32-bit floating point precision. Throughput can be increased by using a larger batch size (at the cost of higher latency), and/or reducing bitwidths through quantization., with active power consumption under 10 W. The attained throughput is an order of magnitude higher than prior work on depth estimation, with only a slight loss of accuracy; FastDepth achieves a $\delta_{1}$ accuracy222Accuracy metrics are defined in Section IV-A. of 77.1% on NYU Depth v2.

The low latency and high throughput attainable with our network is motivated by practical robotic systems, where multiple programs (such as localization, mapping, motion planning, control, and potentially other perception tasks) all run in parallel. Each program demands a certain amount of computational resources, and consequently, the CPU/GPU is not dedicated to the depth estimation task alone. A lower latency network design enables real-time performance even with a limited computing budget.

In summary, this paper demonstrates real-time monocular depth estimation using a deep neural network that achieves the lowest latency and highest throughput on an embedded platform that can be carried by a micro aerial vehicle.

II Related Work

In this section, we summarize past research done on depth estimation, efficient neural networks, and network pruning.

II-A Monocular Depth Estimation

Depth estimation from a single color image has been an active research topic in both the robotics and computer vision communities for over a decade. Early works on depth estimation using RGB images captured by a monocular camera usually relied on hand-crafted features and probabilistic graphical models. For instance, Saxena et al. [6] estimated the absolute scales of different image patches and inferred depth using a Markov Random Field model. Non-parametric approaches [7, 8, 9, 10] were also exploited to estimate the depth of a query image by combining the depths of images with similar photometric content retrieved from a database. Since then, depth estimation has evolved from using simple handcrafted feature representations [6] to modern deep learning based approaches [11, 12, 13, 14].

State-of-the-art RGB-based depth estimation methods use deep learning based methods to train a convolution neural network using large-scale datasets [15, 16, 12]. Eigen et al. [11] suggested a two-stack convolutional neural network (CNN), with one stack predicting the global coarse scale and the other stack refining local details. Eigen and Fergus [16] further incorporated other auxiliary prediction tasks into the same architecture. Liu et al. [15] combined a deep CNN and a continuous conditional random field, and attained visually sharper transitions and local details. Laina et al. [12] developed a deep residual network based on ResNet [17] and achieved higher accuracy than [15, 16]. Qi et al. [18] trained networks to estimate both the depth and the normals, as a way to address the problem of blurriness in predictions. Semi-supervised [19] and unsupervised learning [20, 21, 22] setups have also been explored for disparity image prediction. For instance, Godard et al. [22] formulated disparity estimation as an image reconstruction problem, where neural networks were trained to warp left images to match the right. Mancini et al. [23] proposed a CNN that took both RGB images and optical flow images as input to predict distance. Fusion of RGB images and sparse depth measurements [24, 25] at early stages also improved the accuracy of depth estimation.

All of these methods focus heavily on attaining higher accuracy at increased complexity and runtime cost, with diminishing accuracy improvement. For instance, the $\delta_{1}$ accuracy of depth estimation on the NYU Depth V2 dataset [26] saturates at around 82% in recent years [12, 18].

II-B Efficient Neural Networks

There has been significant effort in prior work to design efficient neural networks. As an example, for image classification, MobileNet [2] achieves similar accuracy as VGG-16 [27] but has $2.7$ times fewer multiply-and-accumulate operations (MACs) and $32.9$ times fewer weights. For object detection, SSD [28] is $6.6$ times faster than Faster-RCNN [29] with a higher mean average precision (mAP).

However, most previous work in this space has focused on encoder networks that reduce an input image into a label. Designing efficient neural networks for applications requiring pixel-based results, where an encoder network is followed by a decoder network, has been less explored. As will be shown in Figure 3, in existing designs that achieve close to state-of-the-art accuracy on the depth estimation task, the decoder largely dominates inference runtime. In our work, we emphasize efficient encoder-decoder network design. In particular, our usage of depthwise separable convolution in the decoder differentiates us from existing approaches and enables us to develop an architecture in which the decoder no longer dominates inference runtime.

II-C Network Pruning

Hand-crafted networks are usually over-parameterized due to the ease of training, which reduces network efficiency. To address this problem, network pruning, such as [30, 31, 32, 3], is widely used to identify and remove redundant parameters and computation. However, these pruning methods are mainly applied on encoder networks. In this work, we adopt a state-of-the-art algorithm, NetAdapt [3], to demonstrate how pruning can improve the efficiency of both encoder and decoder networks used in a depth estimation network design.

III Methodology

In this section, we describe our proposed network architecture, the motivation behind our design choices, and the steps we take to reduce inference runtime.

III-A Network Architecture

Our proposed fully convolutional encoder-decoder architecture is shown in Figure 2. The encoder extracts high-level low-resolution features from the input image. These features are then fed into the decoder, where they are gradually upsampled, refined, and merged to form the final high-resolution output depth map. In developing a depth estimation network that can run in real-time, we seek low-latency designs for both the encoder and the decoder.

III-A1 Encoder Network

The encoder used in depth estimation networks is commonly a network designed for image classification. Popular choices include VGG-16 [27] and ResNet-50 [17] because of their strong expressive power and high accuracy. However, such networks also suffer from high complexity and latency, making them unsuitable for applications running in real-time on embedded systems.

Targeting low latency, we employ a state-of-the-art efficient network, MobileNet [2], as our encoder of choice. MobileNet makes use of depthwise decomposition, which factorizes an $m{\times}m{\times}n$ standard convolutional layer into $n$ $m{\times}m$ depthwise layers and a $1{\times}1$ pointwise layer. Since each filter in a depthwise layer only convolves with a single input channel, the complexity of a depthwise layer is much lower than that of a standard convolutional layer, where each filter convolves with all input channels. Moreover, each pointwise filter is just a $1{\times}1$ kernel, so the number of MACs performed by a pointwise layer is $m^{2}$ times smaller than that of the original standard convolution. Therefore, depthwise decomposition significantly reduces the complexity of a convolutional layer, making MobileNet more efficient than networks with standard convolution like ResNet and VGG. This translates to reduced latency,

III-A2 Decoder Network

The objective of the decoder is to merge and upsample the output of the encoder to form a dense prediction. A key design aspect of the decoder is the upsample operation used (e.g., unpooling, transpose convolution, interpolation combined with convolution).

Our decoder network (termed NNConv5) consists of five cascading upsample layers and a single pointwise layer at the end. Each upsample layer performs $5\times 5$ convolution and reduces the number of output channels by $1/2$ relative to the number of input channels. Convolution is followed by nearest-neighbor interpolation that doubles the spatial resolution of intermediate feature maps. Interpolating after convolution instead of before lowers the resolution of feature maps processed by the convolutional layers. We use depthwise decomposition to further lower the complexity of all convolutional layers, resulting in a slim and fast decoder.

III-A3 Skip Connections

Encoder networks typically contain many layers to gradually reduce the spatial resolution and extract higher-level features from the input. The output of the encoder into the decoder becomes a set of low resolution features in which many image details can be lost, making it more difficult for the decoder to recover pixel-wise (dense) data. Skip connections allow image details from high resolution feature maps in the encoder to be merged into features within the decoder; this helps the decoding layers reconstruct a more detailed dense output. Skip connections have been previously been used in networks for image segmentation such as U-Net [33] and DeeperLab [34], showing that they can be beneficial in networks producing dense outputs.

We incorporate skip connections from the MobileNet encoder to the outputs of the middle three layers in the decoder. Feature maps are combined with via addition rather than concatenation, to avoid increasing the number of feature map channels processed by the decoding layers.

III-B Network Compilation

Our proposed network architecture is fully convolutional and makes use of depthwise decomposition in both the encoder and the decoder. Depthwise separable convolutional layers are currently not yet fully optimized for fast runtime in commonly-used deep learning frameworks. This motivates the need for hardware-specific compilation to translate the complexity reduction achievable with depthwise layers into runtime reduction on hardware. We use the TVM compiler stack [4] to compile our proposed network design for deployment on embedded platforms such as the Jetson TX2.

III-C Network Pruning

To reduce network latency even further, we perform post-training network pruning using the state-of-the-art algorithm, NetAdapt [3]. Starting from a trained network, NetAdapt automatically and iteratively identifies and removes redundant channels from the feature maps to reduce the computational complexity. In each iteration, NetAdapt generates a set of network proposals simplified from a reference network. The network proposal with the best accuracy-complexity trade-off is then chosen and used as the reference network in the next iteration. The process continues until the target accuracy or complexity is achieved. Network complexity can be gauged by indirect metrics (e.g., MACs) or direct metrics (e.g., latency on a target hardware platform).

IV Experiments

In this section, we present experiment results to demonstrate our approach. We first present an evaluation against existing work and then provide an ablation study of our design. We offer comparisons of various encoder and decoder options, analysing them based on accuracy and latency metrics. We also show that hardware-specific compilation reduces the runtime cost of the depthwise separable layers within our network and that network pruning helps improve the efficiency of both of the encoder and the decoder.

IV-A Experiment Setup

We train our networks and evaluate their accuracy on the NYU Depth v2 dataset [26] with the official train/test data split. Network training is similar to [24] and is implemented in PyTorch [35] with 32-bit floating point precision. For training, a batch size of 8 and a learning rate of $0.01$ are used. The optimizer is SGD with a momentum of 0.9 and a weight decay of 0.0001. Encoder weights (e.g., for MobileNet and ResNet) are pretrained on ImageNet [36]. Accuracy is measured by both $\delta_{1}$ (the percentage of predicted pixels where the relative error is within $25\%$) and RMSE (root mean squared error). For evaluation, a batch size of 1 is used. Precision is kept at 32-bit floating point. Our primary target platform is a Jetson TX2 in max-N mode.333Max-N mode: all CPU cores in use and GPU clocked at 1.3 GHz.

IV-B Final Results and Comparison With Prior Work

Results achieved with our methodology are summarized in Figure 3. ResNet-50 with UpProj serves as a baseline; this network follows the architecture described in [12].444With modifications, namely a $224\times 224$ input to the encoder and five (instead of four) upsample layers in the decoder. This is done to match our architecture and allow for a more fair comparison to our network. Consequently, the accuracy and runtime reported in Figure 3 will differ from that of the unmodified model reported as part of our evaluation in Table I. The runtime of this baseline network is largely dominated by the decoder. In our approach, the most immediate and significant runtime reduction comes from using a smaller and computationally simpler decoder. However, the decoder continues to dominate network runtime when combined with the MobileNet encoder. By simplifying the convolutions within the decoder with depthwise decomposition, the runtime of the decoder begins to more closely match that of MobileNet. Pruning and compiling the network for the target Jetson TX2 platform lower the runtime of both the encoder and the decoder even further, ultimately reducing total inference runtime by a factor of 65 times relative to the baseline and enabling an extremely high throughput of up to 178 fps.

Accuracy and latency metrics of our model in comparison with prior work555MACs and runtimes were generated from our re-implemented models. are summarized in Table I. We evaluate against depth estimation methods that use deep learning but do not involve additional processing such as CRFs, since the additional processing incurs computation and runtime cost. We measure the active power consumption when running our model on the TX2 in max-N mode to be under 10 W. We additionally report runtime and power consumption data for the TX2 in the more energy-efficient max-Q mode.666Max-Q mode: only the ARM Cortex-A57 cores in use and GPU clocked at 0.85 GHz. Configured for best power-throughput tradeoff.  Table II summarizes this data. We note that with the TX2 in max-Q mode, our model can achieve close to real-time speeds on the CPU and can still easily surpass real-time speeds on the GPU, with active power consumption under 5 W.

Figure 4 visualizes results of depth estimation produced by our model on images from the NYU Depth v2 dataset. Skip connections between encoding and decoding layers improve the sharpness and visual clarity of depth map outputs, while network pruning preserves and even enhances clarity. We also show an error map visualizing the difference between the output of our model and ground truth, noting that the error is highest at boundaries and at distant objects.

IV-C Ablation Study: Encoder Design Space

A common encoder used in existing high-accuracy approaches [12, 37] is ResNet-50 [17]. Targeting lower encoder latency, we consider the smaller ResNet-18 [17] and MobileNet [2] as alternatives to ResNet-50. The last average pooling layer and fully connected layers are removed from the MobileNet and ResNet architectures. To make the encoders compatible with a fixed decoder structure, we append a $1{\times}1$ convolutional layer to the end of ResNet encoders, such that the output from all encoder variants has a consistent shape of $7{\times}7$ with $1024$ channels.

We compare all three encoder options against each other in Table III. The reported runtimes are obtained by compiling and running the encoder networks in PyTorch. Runtimes for ResNet-50 and ResNet-18 are too high, even on the TX2 GPU, to achieve real-time speeds (i.e., above 30 fps) if these encoders are paired with decoders of similar latency. In comparison, MobileNet efficiently trades off between accuracy and latency, and has a noticeably lower GPU runtime. We therefore select MobileNet as our encoder.

We note that despite its lower complexity, MobileNet is an order of magnitude slower on the TX2 CPU than ResNet-18. This can be attributed to as-of-yet unoptimized implementations for depthwise layers in deep learning frameworks, motivating the need for an alternate deep learning compiler, as will be discussed in Section IV-E.

IV-D Ablation Study: Decoder Design Space

While encoders have been well characterized in deep learning applications, decoders have been less extensively explored, especially in the context of efficient network design. We consider several decoder design aspects: upsample operation, depthwise decomposition, and skip connections.

IV-D1 Upsample Operation

We survey four ways of upsampling in the decoder. Their characteristics are listed below, and visual representations are shown in Figure 5:

1. (a)

UpProj [12]: $2\times 2$ unpooling (zero-insertion) followed by a two-branched residual structure that computes a total of three convolutions (two $5\times 5$ and one $3\times 3$). 2. (b)

UpConv [12]: $2\times 2$ unpooling (zero-insertion) followed by a single 5$\times$5 convolution. 3. (c)

DeConv5: transpose convolution using a $5\times 5$ kernel.777We use a kernel size of 5 to fairly compare against UpConv. 4. (d)

NNConv5: $5\times 5$ convolution followed by nearest-neighbor interpolation888An alternate option would be bilinear interpolation. We select nearest-neighbor interpolation as is it a simpler operation with more consistent implementations across deep learning frameworks and compilers. with a scale factor of 2.

We implement four decoder variants using these upsample operations, keeping the structure fixed at 5 decoding layers with $1\times 1$ convolution at the end. Table IV compares the four decoders. UpProj is most complex, due to its larger number of convolutions per upsample layer. It achieves the highest $\delta_{1}$ accuracy but is the slowest. UpConv is less complex and faster than UpProj, but its CPU and GPU runtimes are still too slow for real-time processing. DeConv5 has an identical number of weights and MACs as UpConv and is noticeably faster on both the CPU and GPU. However, it can be prone to introducing checkerboard artifacts in its outputs [38], which helps explain its lower accuracy. NNConv5 achieves higher $\delta_{1}$ accuracy and lower RMSE than both UpConv and DeConv5, with a slightly lower GPU runtime. We therefore select NNConv5 as our decoder.

IV-D2 Depthwise Separable Convolution

After selecting MobileNet as our encoder and NNConv5 as our decoder, we observe that the runtime of our network is dominated by the decoder (see Figure 3 for the encoder-decoder breakdown). This motivates us to simplify our decoder even further. Similar to how depthwise decomposition lowers the complexity and latency in MobileNet, we now replace all convolutions within the decoder with depthwise separable convolutions.

Table V shows that depthwise decomposition in the decoder lowers inference runtime on the GPU by almost half.999In contrast, runtime on the CPU increases, despite reduced number of MACs. This is due to the inefficient CPU operation of depthwise layers. However, as was the case with MobileNet, depthwise layers in the decoder result in a slight accuracy loss, due to the reduction in trainable parameters and computation. In order to restore some of the lost accuracy, we incorporate skip connections between the encoding and decoding layers.

IV-D3 Skip Connections

We consider both additive and concatenative skip connections. Concatenative skip connections increase the computational complexity of the decoder since decoding layers need to process feature maps with more channels.  Table V shows that this improves the $\delta_{1}$ accuracy but also noticeably increases CPU and GPU runtimes. In contrast, using additive skip connections leaves the number of channels in the decoder unchanged and has a negligible impact on inference runtime while achieving almost the same accuracy boost. We therefore use additive skip connections in our final network design. As shown in Figure 4(d), skip connections noticeably improve the sharpness and visual clarity of the depth maps output by our network design.

IV-E Hardware-Specific Optimization

Current deep learning frameworks rely on framework-specific operator libraries, where the level of hardware-specific optimization of operator implementations may vary. Our proposed network architecture incorporates depthwise layers throughout the encoder and decoder. These layers are currently not yet fully optimized in commonly-used deep learning frameworks. As a result, although depthwise decomposition significantly reduces the number of MACs in a network, a similar reduction is not reflected in latency. The left portion of Table VI highlights exactly this: the TX2 CPU runtime of MobileNet-NNConv5 is high to begin with, due to the prevalence of depthwise layers in MobileNet, and it increases even more when we use depthwise layers in the decoder. To address the observed runtime inefficiencies of depthwise layers, we use the TVM compiler stack [4]. TVM performs hardware-specific scheduling and operator tuning that allows the impact of reduced operations to be translated into reduced processing time. The right portion of Table VI reports TX2 runtimes for networks compiled with TVM. Depthwise decomposition in the decoder now reduces CPU runtime by 3.5 times and GPU runtime by 2.5 times.

IV-F Network Pruning

Prior to network pruning, our architecute (MobileNet-NNConv5 with depthwise decomposition in the decoder and additive skip connections) already surpasses real-time throughput on the TX2 GPU but does not yet achieve real-time speeds on the TX2 CPU. Network pruning lowers the model’s runtime so that it can achieve a CPU framerate above 25 fps that is more suitable for real-time inference. As shown in  Table VII, pruning achieves a 2 times reduction in MACs, a 1.5 times reduction in GPU runtime, and a 1.8 times reduction in GPU runtime with almost the same accuracy.  Figure 4(e) shows that pruning process preserves the sharpness and visual clarity of output depth maps.

Fig. 6 shows the pruned architecture. We can see that there are two bottlenecks: one in the encoder (the layer mobilenet.9) and one in the decoder (the layer decoder.2). This is consistent with the observations in [3, 39].

V Conclusion

In this work, we enable high-speed depth estimation on embedded systems. We achieve high frame rates by developing an efficient network architecture, with a low-complexity and low-latency decoder design that does not dominate inference runtime even when combined with a small MobileNet encoder. The size of our compact model is further reduced by applying a state-of-the-art pruning algorithm. Hardware-specific compilation is used to translate complexity reduction into lower runtime on a target platform. On the Jetson TX2, our final model achieves runtimes that are an order of magnitude faster than prior work, while maintaining comparable accuracy.

Although this work focuses on depth estimation, we believe that similar approaches can be used to achieve real-time performance with deep-learning based methods for other dense prediction tasks, such as image segmentation.