Indoor dense depth map at drone hovering

TL;DR

This paper presents a novel method for reconstructing dense depth maps in indoor environments for drone navigation, utilizing sparse point clouds from vSLAM and local plane fitting techniques during drone hovering.

Contribution

It introduces a patch-based local plane fitting approach combined with a plane sweep technique, improving dense depth estimation in low-textured indoor scenes during small drone motions.

Findings

Better depth reconstruction in artificial lighting conditions

Improved performance in low-textured environments

Effective during small camera motions

Abstract

Autonomous Micro Aerial Vehicles (MAVs) gained tremendous attention in recent years. Autonomous flight in indoor requires a dense depth map for navigable space detection which is the fundamental component for autonomous navigation. In this paper, we address the problem of reconstructing dense depth while a drone is hovering (small camera motion) in indoor scenes using already estimated cameras and sparse point cloud obtained from a vSLAM. We start by segmenting the scene based on sudden depth variation using sparse 3D points and introduce a patch-based local plane fitting via energy minimization which combines photometric consistency and co-planarity with neighbouring patches. The method also combines a plane sweep technique for image segments having almost no sparse point for initialization. Experiments show, the proposed method produces better depth for indoor in artificial lighting condition, low-textured environment compared to earlier literature in small motion.

Figures

Equations

$$\begin{gathered} C(\zeta )=\sum _{p\in \omega }{\lambda }_p{C}_p({\Psi }_p^D+{\Psi }_p^I+{\lambda }_g{\Psi }_p^G) \\ +\tau \sum _{(p,q)\in {\mathcal{N}}_{\omega }}{\lambda }_{pq}{C}_p{\Psi }_{pq}^C \end{gathered}$$

$${\lambda }_p=\sigma {\beta }_p$$

$${\lambda }_{pq}=\frac{1}{\xi }\sum _{x\in \xi }\delta (x)$$

$$C_p=\{\begin{array}{ll}0& \text{: if (η∗area(φp)) ≥ Δ}\\ \frac{1}{\eta \ast area({\phi }_p)}& \text{: η ≥ 0}\\ 1& \text{: Otherwise}\end{array}$$

$${\Psi }_p^D=\sum _{i\in {\phi }_p}\frac{1}{\partial }(\Vert X_i-X_{i{\zeta }_p}\Vert )$$

$${\pi }_p(\upsilon )={\lambda }_{\upsilon }H(p,\{{\nu }_p,{\vec{n}}_p\})$$

$${\Psi }_p^I=\sum _{p\in \omega }\sum _{x\in \theta }Var(I_{1{\zeta }_p},\cdots ,I_{\upsilon {\zeta }_p})$$

$${\Psi }_{pq}^C=\{\begin{array}{ll}0& \text{:for connected surfaces (1st \& 2nd rows)}\\ {\varrho }_1& \text{:for disconnected surfaces (3rd row)}\\ {\varrho }_2& \text{:for ocluded surfaces (4th row)}\\ {\varrho }_3& \text{:otherwise}\end{array}$$

Full text

**Index Terms— ** Small baseline, Depth propagation, Indoor Reconstruction, low-textured environment.

1 Introduction

Autonomous navigation of Micro Aerial Vehicles (MAVs) in an outdoor cluttered environment is a well-researched topic in robotic research on last decade [4, 5, 6, 7] but ignored for indoor navigation due to low-textured man-made environments. Autonomous navigation of MAVs requires robust estimation of a robot’s pose along with dense depth for navigable space detection. Visual Simultaneous Localization and Mapping (vSLAM) address the problem of estimating camera pose along with 3D scene structure and it achieved significant improvement [8]. Most of the existing vSLAMs produce a sparse 3D structure which is insufficient for navigable space detection but executing a real-time dense vSLAM is computationally very heavy. In contrast to a dense SLAM, dense depth map computation during hovering of a drone for locating free space is computationally less complex. After understanding the depth map drone can start moving using sparse vSLAM until it requires another understanding for new free space and hover again. While the drone is hovering, the baselines between consecutive images are very small, therefore, we need to estimate the depth map using the structure from Small Motion (SfSM) [9] which has the advantages like better photometric constraints, rotation matrix simplification, etc. [9] over regular motion. In this paper, we present a novel dense depth map estimation using SfSM at drone hovering.

Traditional feature based vSLAM approaches [10, 11, 12, 1] produce erroneous metric depth when the baseline is less than $\sim$8 mm and viewing angle of a 3D point is less than 0.2*∘* [9]. Despite several works on SfSM [9, 13, 14, 2, 3, 15, 16], there exists inaccuracy in the estimation of dense depth for navigable space detection on the fly. Im et al. [14] present a SfSM system that uses a plane fitting using colour and geometry consistency. Im et al. propose another SfSM algorithm [2] using plane fitting through a quadratic energy function consists of sparse data points, colour and geometric smoothness. Both the systems [14, 2] use sparse 3D points for geometric validation, which is erroneous in low-textured indoor environments due to less number of features. Both the systems [14, 2] fit a continuous plane in the entire visible 3D space which fails to estimate sudden depth variation. Fig. 1(a), (b) show such erroneous depth estimation by [2]. Recently, Ha et al. present a SfSM system [3] that uses a plane sweep technique which produces discrete dense depth and fails to estimate reliable depth for regular planar surfaces along with edge boundaries. The plane sweep runs on image pixels without considering the sparse point position which allows a pixel to attain a particular depth without any neighbouring constrain and estimate noisy depth on a planar surface as shown in Fig. 1(c). The method is not real-time (takes around 10 minutes). Javidnia and Corcoran present a ORB feature [17] based SfSM system [15] where plane sweeping technique generates dense depth directly from ORB feature matches. The ORB feature highly depends on texture present on the scene, therefore the depth map from [15] is also erroneous in the low-textured environment. The system running time is about 8 minutes claimed by the authors. So there exists no system which produces reliable dense depth for free space estimation in indoor.

In this work, we target to obtain an accurate dense depth map for navigable space detection by a drone in indoor environments using small motion configuration [9]. We use Edge SLAM [1], the most resilient SLAM in low-textured environment, for estimating camera poses and initial sparse 3D points while the drone is in normal motion and allow the drone to hover and continue estimating the small motion camera poses using resectioning [18]. We start the dense depth estimation as soon as we get at least 20 estimated images from Edge SLAM with small motion where the first image in small motion is the reference image. We introduce a segmentation step at the beginning in contrast with previous approaches [14, 2, 3, 15], to segment the reference image into smaller regions to avoid erroneous depth estimation through continuous plane fitting. Segmentation was never been used in the literature of SfSM and this is a contribution in this work. Depth propagation runs on every image segment using a novel patch-based (a minimal segmented 2D area which can be considered as a planar surface in 3D) plane fitting approach through minimizing a cost function consisting of sparse 3D points, intensity and gradients of pixels and co-planarity with neighbouring patches. We design the patch-based cost function for each image segment independently that produces a better depth estimation compared with earlier approaches (e.g. Fig. 1), which is the major contribution of this work. We attach a weight with every patch based on its depth initialization and further add that patch for optimization when the weight is below to certain threshold. This novel idea of weighted patch-based optimization is computationally lightweight which makes the method suitable for real-time systems like drone navigation and we achieve $\sim$14 seconds in execution time to estimate dense depth using 20 images.

Section 2 illustrates the insights of our proposed method. Section 3 provides the experimental results. Section 4 concludes this paper.

2 Proposed Methodology

2.1 Segmentation

We segment the sparse point cloud using a combination of nearest neighbour clustering and colour similarity between 3D points based on [19]. We use a Kd-tree representation of input sparse point cloud. The clustering algorithm generates clusters of points having close location proximity (lower than an adaptive proximity threshold based on camera locations) with similar colour. Fig. 2(b) show the segmented point cloud of the Lab-room dataset where the floor, walls, objects belongs to separate segments.

We apply a non-linear bilateral filter [20] to de-noise the reference image. The basic idea of bilateral filtering is to do an additional filtering in range domain along with traditionally filtering in the spatial domain. We use a bilateral filter to preserve the edge properties of the image as it combines the pixel intensities based on their geometric as well as photometric closeness. Further, we segment the filtered image using a colour based region growing segmentation based on [21]. We further use the segmented sparse 3D point clusters to merge 2D segments. 3D points present in a cluster should lie on the same image segment. We project the 3D points in a cluster on the image plane and merge the smaller image segments based on spatial information (e.g. Fig. 2(c)). Our depth propagation is erroneous only if the segmentation fails both in 2D and 3D. But such cases occur only when an entire segment is of similar colour and no sparse point present to differentiate abrupt depth change in 3D but the occurrence of such situations is very infrequent.

2.2 Depth Propagation

Depth propagation method estimates the dense depth for every image segment in patch-based. An image segment becomes a set of patches $\omega$ and another set of neighbouring patches $\mathcal{N_{\omega}}\ \forall\ p\in\omega$. Every patch corresponds to a planar 3D surface characterized as ${\zeta}_{p}=\{\nu_{p},\vec{n}_{p}\}$. $\nu_{p}$ represents a point on ${\zeta}_{p}$ and $\vec{n}_{p}$ represents the normal vector of ${\zeta}_{p}$ from $\nu_{p}$. $\theta$ is the set of pixels in patch $p$, ${\varphi}_{p}$ represents the set of sparse 3D points whose projection lies within the patch $p$ boundary and $\eta=|{\varphi}_{p}|$. We initialize all the patches $p\in\omega$ with a planar surface in the most feasible way. We start with the patch that has largest $\eta$. Initialize it by fitting a 3D planar surface using the points in ${\varphi}_{p}$ having reprojection error $<0.1$ pixel. Subsequently, continue the initialization for neighbouring patches where $\eta>0$ using a similar plane fitting technique. There still exist uninitialized patches where $\eta=0$ and we initialize them by connecting the 3D surfaces of already initialized neighbouring patches. There may exist image segments where $\eta=0\ \forall\ p\in\omega$ and initialization fails. We calculate the gradient of every patch by adding intensity gradients of all pixels belong to the patch $p$. We use the plane sweep approach as proposed in [3] only for the patch with largest gradient. Plane sweep runs a virtual plane perpendicular to the viewing direction from maximum to minimum depths obtained from sparse point cloud. The particular depth with minimum photometric error initializes the dense depth for that patch. Subsequently, continue the initialization for neighbouring patches using continuous plane fitting. We formulate the cost $C(\zeta)$ as in Eq. 1 and minimize it through the parameters $\nu_{p}$ and $\vec{n}_{p}$.

[TABLE]

where ${\lambda}_{p}$ and ${\lambda}_{pq}$ represent adaptive normalizing weights. $C_{p}$ is a confidence weight for every patch $p$. ${\lambda}_{g}$ represents the weight for ${\Psi}_{p}^{G}$ and $\tau$ represents a balancing factor between the data term and the regularization term.

[TABLE]

where ${\sigma}{\beta}_{p}$ represents the variation in projected area of 2D patches using the points in $\theta$ and all views $\upsilon\in V$.

[TABLE]

where $\xi$ represents the set of edge pixels between $p$ and $q$. $\delta(x)$ is the intensity gradient of the pixels $x\in\xi$. We design $C_{p}$ as in Eq. 4, which speeds up the optimization by providing lower weight to stable patches.

[TABLE]

where $area({\varphi}_{p})$ represents the 2D projected area of points in ${\varphi}_{p}$ and $\Delta$ is a given threshold. A patch with $C_{p}=0$ is considered as stable because $(\eta*area({\varphi}_{p}))\geq\Delta$ and therefore excluded in optimization. Fig. 3 shows a comparison result where $C_{p}=0$ for 258 patches, $C_{p}<1$ for 636 patches and $C_{p}=1$ for 1865 patches in the image segment. Our weighted patch-based optimization performs $\sim$9.7 times faster compared to full optimization. The full optimization reduces some artefacts (shown in inset) with a smoothness on the surface, but this little improvement has almost no impact on free space calculation.

Sparse 3D point consistency: We formulate the sparse point consistency ${\Psi}_{p}^{D}$ as:

[TABLE]

where $\partial$ denotes the average reprojection error for the point $i\in{\varphi}_{p}$. $X_{i}$ & $X_{i{\zeta}_{p}}$ denote the sparse depth and the depth on the surface ${\zeta}_{p}$ for the point $i$. ${\Psi}_{p}^{D}$ provides lower weight to the sparse 3D points having high reprojection error.

Photo-consistency: The term ${\Psi}_{p}^{I}$ in Eq. 1 represents the cost of intensity matching between a patch $p$ and it’s warped patches $\pi_{p}(\upsilon)$ in other views $\upsilon\in V$ using plane-induced homography [22] in small baseline. $\pi_{p}(\upsilon)$ is designed as:

[TABLE]

where $H$ is the homography matrix [22] of surface ${\zeta}_{p}$ on the view $\upsilon$, ${\lambda}_{\upsilon}$ is the weight for view $\upsilon$ which provides more priority to nearer frames because intensity variation is less sensitive to closer views with depth variation. We design the intensity cost as:

[TABLE]

where $Var(\cdotp)$ is the variance function and $I_{i{\zeta}_{p}}$ represents the intensity of the pixel $x\in\theta$ in view $\upsilon$ projected from surface ${\zeta}_{p}$. We define a more robust variance calculation by providing higher priority to images nearer to the reference image. The term ${\Psi}_{p}^{G}$ in Eq. 1 represents the cost of gradient matching which follows the same variance formula as described in Eq. 7 with the gradient of intensity at the place of intensity.

Regularization: The pairwise regularization works on the assumption of connectivity with neighbouring patches at edge boundaries. We consider five types of priority based patch pair configuration to cater all possible occupancy conditions. Fig. 4(a) shows four such pairs refer in Eq. 8.

[TABLE]

where $0<{\varrho}_{1}<{\varrho}_{2}<{\varrho}_{3}$

3 Experimental Results

We use an Intel i7-8700 (6 cores @3.7-4.7GHz) with 16Gb RAM, for implementation and a Bebop quadcopter from Parrot Corporation for data acquisition. We consider registered Kinect point cloud as ground truth as shown in Fig. 1. We perform all our experiments on 640x480 VGA resolution. All experiments use parameters: $\Delta=7$, ${\lambda}_{g}=3$, $\tau=1.7$, ${\varrho}_{1}=0.6$, ${\varrho}_{2}=3.5$, ${\varrho}_{3}=20$.

We present two indoor datasets suitable for drone navigation to evaluate the performance of our proposed method in comparison with earlier methods. Sparse reconstruction in all the cases are executed using Edge SLAM [1]. Fig. 4(b) shows Office-Space dataset, an indoor office premise with smooth depth variation along the wall and a sudden depth changes at wall boundary. We compare our result with Kinect and the results obtained using [2] and [3]. We show a region with sudden depth variation (3rd column) and another region with a smooth planar surface (4th column). Both the earlier methods [2, 3] fail to estimate sudden depth changes at object boundary and erroneous depth estimation on the planar surface.

due to continuous plane fitting by [2] and erroneous plane sweep by [3] as already explained in Sec. 1. The mean error in depth estimation using our method against Kinect point cloud is 0.2352 meter whereas method by [3] producing 2.11575 meters errors. Fig. 4(c) represents a Corridor dataset, a corridor with an object on the ground. Our estimation shows better accuracy in estimating depth variation on the wall as well as on the ground compared to [2, 3] for the similar reasons. Our dense depth estimation has least artefacts and more realistic for free space understanding in both cases.

Execution Time: Timing is a vital factor for any method to run on the drone. Our method achieves a running time of $\sim$14 sec using 20 images on our unoptimized implementation.

4 Conclusion

We have demonstrated an approach to estimate indoor dense depth map on drone hovering using camera calibration and sparse point cloud from Edge SLAM. We have introduced a novel segmentation procedure at beginning to segment the reference image in objects and estimate depth independently for each segment which improves the accuracy in depth estimation. We have proposed a novel patch-based plane fitting approach for depth estimation through minimizing a cost function which consists of sparse points, photo consistency and regularization terms. Our method attended a significant improvement in term of accuracy and feasibility for any real-time platform like drones.