PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction

TL;DR

PlaneMatch introduces a deep learning-based descriptor for detecting coplanar surfaces in RGB-D SLAM, improving robustness and accuracy in 3D reconstruction by leveraging learned coplanarity constraints.

Contribution

We propose a novel deep neural network descriptor for coplanarity detection in RGB-D data, trained on a large dataset, and demonstrate its effectiveness in improving SLAM reconstruction.

Findings

Outperforms existing methods in coplanarity detection accuracy.

Enables more robust RGB-D reconstruction with coplanarity constraints.

Achieves comparable or superior results to state-of-the-art frameworks.

Abstract

We introduce a novel RGB-D patch descriptor designed for detecting coplanar surfaces in SLAM reconstruction. The core of our method is a deep convolutional neural net that takes in RGB, depth, and normal information of a planar patch in an image and outputs a descriptor that can be used to find coplanar patches from other images.We train the network on 10 million triplets of coplanar and non-coplanar patches, and evaluate on a new coplanarity benchmark created from commodity RGB-D scans. Experiments show that our learned descriptor outperforms alternatives extended for this new task by a significant margin. In addition, we demonstrate the benefits of coplanarity matching in a robust RGBD reconstruction formulation.We find that coplanarity constraints detected with our method are sufficient to get reconstruction results comparable to state-of-the-art frameworks on most scenes, but outperform other methods on standard benchmarks when combined with a simple keypoint method.

Figures

Tables

Table 1. (a) Comparison to alternatives.

Method	fr1/desk	fr2/xyz	fr3/office	fr3/nst
RGB-D SLAM	2.3	0.8	3.2	1.7
VoxelHashing	2.3	2.2	2.3	8.7
Elastic Fusion	2.0	1.1	1.7	1.6
Redwood	2.7	9.1	3.0	192.9
Fine-to-Coarse	5.0	3.0	3.9	3.0
BundleFusion	1.6	1.1	2.2	1.2
Ours	1.4	1.1	1.6	1.5
BundleFuison+Ours	1.3	0.8	1.5	0.9

Table 2. (a) Comparison to alternatives.

Method	fr1/desk	fr2/xyz	fr3/office	fr3/nst
RGB-D SLAM	2.3	0.8	3.2	1.7
VoxelHashing	2.3	2.2	2.3	8.7
Elastic Fusion	2.0	1.1	1.7	1.6
Redwood	2.7	9.1	3.0	192.9
Fine-to-Coarse	5.0	3.0	3.9	3.0
BundleFusion	1.6	1.1	2.2	1.2
Ours	1.4	1.1	1.6	1.5
BundleFuison+Ours	1.3	0.8	1.5	0.9

Table 3. (b) Comparison to baselines.

Method	fr1/desk	fr2/xyz	fr3/office	fr3/nst
No D. in RANSAC	9.6	4.8	12.6	2.3
No D. in Opt.	4.8	2.7	2.5	1.9
No D. in Both	18.9	8.3	16.4	2.4
Key-point Only	5.6	4.4	5.2	2.6
Coplanarity Only	2.5	2.1	3.7	–
Ours	1.4	1.1	1.7	1.5

Equations

$$L_{\text{focal}}(x_a,x_p,x_n)=\max {\left(0,\frac{\alpha -\Delta d_{\text{f}}}{\alpha }\right)}^{\lambda },$$

$$E(T,s)=E_{\text{data-cop}}(T,s)+E_{\text{reg-cop}}(s)+E_{\text{data-kp}}(T,s)+E_{\text{reg-kp}}(s).$$

$$E_{\text{data-cop}}(T,s)=\sum _{(i,j)\in \mathcal{P}}\sum _{\pi \in {\Pi }_{ij}}{w}_{\pi }{s}_{\pi }{\delta }^2({\mathbf{T}}_i,{\mathbf{T}}_j,\pi ),$$

$${\delta }^2=\frac{1}{|{\mathcal{V}}_p|}\sum _{{\mathbf{v}}_p\in {\mathcal{V}}_p}{d}^2({\mathbf{T}}_i{\mathbf{v}}_p,{\varphi }_q^G)+\frac{1}{|{\mathcal{V}}_q|}\sum _{{\mathbf{v}}_q\in {\mathcal{V}}_q}{d}^2({\mathbf{T}}_j{\mathbf{v}}_q,{\varphi }_p^G),$$

$$d({\mathbf{T}}_i{\mathbf{v}}_p,{\varphi }_q^G)=({\mathbf{R}}_i{\mathbf{v}}_p+{\mathbf{t}}_i-{\mathbf{p}}_q)\cdot {\mathbf{n}}_q.$$

$$E_{\text{reg-cop}}(s)=\sum _{(i,j)\in \mathcal{P}}\sum _{\pi \in {\Pi }_{ij}}\mu w_{\pi }\Psi (s_{\pi }),$$

$$E_{\text{data-kp}}(T,s)=\sum _{(i,j)\in \mathcal{P}}\sum _{\theta \in {\Theta }_{ij}}{s}_{\theta }||{\mathbf{T}}_i\mathbf{u}-{\mathbf{T}}_j\mathbf{v}||,$$

$$E_{\text{reg-kp}}(s)=\sum _{(i,j)\in \mathcal{P}}\sum _{\theta \in {\Theta }_{ij}}\mu \Psi (s_{\theta }),$$

$$E(T,s)=E_{\text{data-cop}}(T,s)+E_{\text{reg-cop}}(s)+E_{\text{reg-frm}}(T)$$

$$E_{\text{reg-frm}}(T)=\lambda \sum _{i\in \mathcal{F}}\sum _{\mathbf{v}\in {\mathcal{V}}_i}||{\mathbf{T}}_i\mathbf{v}-{\mathbf{T}}_{i+1}\mathbf{v}|{|}^2,$$

$$\mu (\pi )=\min \{{\mu }_i^{d(p)},{\mu }_j^{d(q)}\},$$

$$\mathcal{E}=\sum_{\mathbf{v}_{p}\in\mathcal{V}_{p}}{\bigl{[}(\mathbf{R}\mathbf{v}_{p}+\mathbf{t}-\mathbf{p}_{q})\cdot\mathbf{n}_{q}\bigr{]}^{2}},$$

$$\mathbf{R}\approx \left(\begin{array}{ccc}1& -r_z& r_y\\ r_z& 1& -r_x\\ -r_y& r_x& 1\end{array}\right),$$

$$\mathcal{E}=\sum _{{\mathbf{v}}_p\in {\mathcal{V}}_p}({\mathbf{v}}_p-{\mathbf{p}}_q)\cdot {\mathbf{n}}_q+\mathbf{r}\cdot ({\mathbf{v}}_p\times {\mathbf{n}}_q)+\mathbf{t}\cdot {\mathbf{n}}_q.$$

$$C=\left[\begin{array}{ccc}{\mathbf{v}}_p^1\times {\mathbf{n}}_q& \cdots & {\mathbf{v}}_p^k\times {\mathbf{n}}_q\\ {\mathbf{n}}_q& \cdots & {\mathbf{n}}_q\end{array}\right]\left[\begin{array}{cc}({\mathbf{v}}_p^1\times {\mathbf{n}}_q{)}^T& {\mathbf{n}}_q\\ ⋮& ⋮\\ ({\mathbf{v}}_p^k\times {\mathbf{n}}_q{)}^T& {\mathbf{n}}_q\end{array}\right].$$

$$\Delta \mathcal{E}=2\left[\begin{array}{cc}\Delta {\mathbf{r}}^T& \Delta {\mathbf{t}}^T\end{array}\right]C\left[\begin{array}{c}\Delta \mathbf{r}\\ \Delta \mathbf{t}\end{array}\right].$$

Full text

1 Introduction

With the recent proliferation of inexpensive RGB-D sensors, it is now becoming practical for people to scan 3D models of large indoor environments with hand-held cameras, enabling applications in cultural heritage, real estate, virtual reality, and many other fields. Most state-of-the-art RGB-D reconstruction algorithms either perform frame-to-model alignment [1] or match keypoints for global pose estimation [2]. Despite the recent progress in these algorithms, registration of hand-held RGB-D scans remains challenging when local surface features are not discriminating and/or when scanning loops have little or no overlaps.

An alternative is to detect planar features and associate them across frames with coplanarity, parallelism, and perpendicularity constraints [3, 4, 5, 6, 7, 8, 9]. Recent work has shown compelling evidence that planar patches can be detected and tracked robustly, especially in indoor environments where flat surfaces are ubiquitous. In cases where traditional features such as keypoints are missing (e.g., wall), there seems tremendous potential to support existing 3D reconstruction pipelines.

Even though coplanarity matching is a promising direction, current approaches lack strong per-plane feature descriptors for establishing putative matches between disparate observations. As a consequence, coplanarity priors have only been used in the context of frame-to-frame tracking [3] or in post-process steps for refining a global optimization [4]. We see this as analogous to the relationship between ICP and keypoint matching: just as ICP only converges with a good initial guess for pose, current methods for exploiting coplanarity are unable to initialize a reconstruction process from scratch due to the lack of discriminative coplanarity features.

This paper aims to enable global, ab initio coplanarity matching by introducing a discriminative feature descriptor for planar patches of RGB-D images. Our descriptor is learned from data to produce features whose L2 difference is predictive of whether or not two RGB-D patches from different frames are coplanar. It can be used to detect pairs of coplanar patches in RGB-D scans without an initial alignment, which can be used to find loop closures or to provide coplanarity constraints for global alignment (see Figure 1).

A key novel aspect of this approach is that it focuses on detection of coplanarity rather than overlap. As a result, our plane patch features can be used to discover long-range alignment constraints (like “loop closures”) between distant, non-overlapping parts of the same large surface (e.g., by recognizing carpets on floors, tiles on ceilings, paneling on walls, etc.). In Figure 1, the two patch pairs shown to the right helped produce a reconstruction with globally flat walls.

To learn our planar patch descriptor, we design a deep network that takes in color, depth, normals, and multi-scale context for pairs of planar patches extracted from RGB-D images, and predicts whether they are coplanar or not. The network is trained in a self-supervised fashion where training examples are automatically extracted from coplanar and noncoplanar patches from ScanNet [10].

In order to evaluate our descriptor, we introduce a new coplanarity matching datasets, where we can see in series of thorough experiments that our new descriptor outperforms existing baseline alternatives by significant margins. Furthermore, we demonstrate that by using our new descriptor, we are able to compute strong coplanarity constraints that improve the performance of current global RGB-D registration algorithms. In particular, we show that by combining coplanarity and point-based correspondences reconstruction algorithms are able to handle difficult cases, such as scenes with a low number of features or limited loop closures. We outperform other state-of-the-art algorithms on the standard TUM RGB-D reconstruction benchmark [11]. Overall, the research contributions of this paper are:

• •

A new task: predicting coplanarity of image patches for the purpose of RGB-D image registration.

• •

A self-supervised process for training a deep network to produce features for predicting whether two image patches are coplanar or not.

• •

An extension of the robust optimization algorithm [12] to solve camera poses with coplanarity constraints.

• •

A new training and test benchmark for coplanarity prediction.

• •

Reconstruction results demonstrating that coplanarity can be used to align scans where keypoint-based methods fail to find loop closures.

2 Related Work

RGB-D Reconstruction: Many SLAM systems have been described for reconstructing 3D scenes from RGB-D video. Examples include KinectFusion [13, 1], VoxelHashing [14], ScalableFusion [15], Point-based Fusion [16], Octrees on CPU [17], Elastic Fusion [18], Stereo DSO [19], Colored Registration [20], and Bundle Fusion [2]. These systems generally perform well for scans with many loop closures and/or when robust IMU measurements are available. However, they often exhibit drift in long scans when few constraints can be established between disparate viewpoints. In this work, we detect and enforce coplanarity constraints between planar patches to address this issue as an alternative feature channel for global matching.

Feature Descriptors: Traditionally, SLAM systems have utlized keypoint detectors and descriptors to establish correspondence constraints for camera pose estimation. Example keypoint descriptors include SIFT [21], SURF [22], ORB [23], etc. More recently, researchers have learned keypoint descriptors from data – e.g., MatchNet [24], Lift [25], SE3-Nets [26], 3DMatch [27], Schmidt et al. [28]. These methods rely upon repeatable extraction of keypoint positions, which is difficult for widely disparate views. In contrast, we explore the more robust method of extracting planar patches without concern for precisely positioning the patch center.

Planar Features: Many previous papers have leveraged planar surfaces for RGB-D reconstruction. The most commmon approach is to detect planes in RGB-D scans, establish correspondences between matching features, and solve for the camera poses that align the corresponding features [29, 30, 31, 32, 33, 34, 35, 36]. More recent approaches build models comprising planar patches, possibly with geometric constraints [4, 37], and match planar features found in scans to planar patches in the models [4, 5, 6, 7, 8]. The search for correspondences is often aided by hand-tuned descriptors designed to detect overlapping surface regions. In contrast, our approach finds correspondences between coplanar patches (that may not be overlapping); we learn descriptors for this task with a deep network.

Global Optimization: For large-scale surface reconstruction, it is common to use off-line or asynchronously executed global registration procedures. A common formulation is to compute a pose graph with edges representing pairwise transformations between frames and then optimize an objective function penalizing deviations from these pairwise alignments [38, 39, 40]. Recent methods [12, 41] use indicator variables to identify loop closures or matching points during global optimization using a least-squares formulation. We extend this formulation by setting indicator variables for individual coplanarity constraints.

3 Method

Our method consists of two components: 1) a deep neural network trained to generate a descriptor that can be used to discover coplanar pairs of RGB-D patches without an initial registration, and 2) a global SLAM reconstruction algorithm that takes special advantage of detected pairs of coplanar patches.

3.1 Coplanarity Network

Coplanarity of two planar patches is by definition geometrically measurable. However, for two patches that are observed from different, yet unknown views, whether they are coplanar is not determinable based on geometry alone. Furthermore, it is not clear that coplanarity can be deduced solely from the local appearance of the imaged objects. We argue that the prediction of coplanarity across different views is a structural, or even semantic, visual reasoning task, for which neither geometry nor local appearance alone is reliable.

Humans infer coplanarity by perceiving and understanding the structure and semantics of objects and scenes, and contextual information plays a critical role in this reasoning task. For example, humans are able to differentiate different facets of an object, from virtually any view, by reasoning about the structure of the facets and/or by relating them to surrounding objects. Both involve inference with a context around the patches being considered, possibly at multiple scales. This motivates us to learn to predict cross-view coplanarity from appearance and geometry, using multi-scale contextual information. We approach this task by learning an embedding network that maps coplanar patches from different views nearby in feature space.

Network Design: Our coplanarity network (Figure 2 and 3) is trained with triplets of planar patches, each involving an anchor, a coplanar patch (positive) and a noncoplanar patch (negative), similar to [42]. Each patch of a triplet is fed into a convolutional network based on ResNet-50 [43] for feature extraction, and a triplet loss is estimated based on the relative proximities of the three features. To learn coplanarity from both appearance and geometry, our network takes multiple channels as input: an RGB image, depth image, and normal map.

We encode the contextual information of a patch at two scales, local and global. This is achieved by cropping the input images (in all channels) to rectangles of $1.5$ and $5$ times the size of the patch’s bounding box, respectively. All cropped images are clamped at image boundaries, padded to a square, and then resized to $224\times 224$. The padding uses 50% gray for RGB images and a value of [math] for depth and normal maps; see Figure 3.

To make the network aware of the region of interest (as opposed to context) in each input image, we add, for each of the two scales, an extra binary mask channel. The local mask is binary, with the patch of interest in white and the rest of the image in black. The global mask, in contrast, is continuous, with the patch of interest in white and then a smooth decay to black outside the patch boundary. Intuitively, the local mask helps the network distinguish the patch of interest from the close-by neighborhood, e.g. other sides on the same object. The global mask, on the other hand, directs the network to learn global structure by attending to a larger context, with importance smoothly decreasing based on distance to the patch region. Meanwhile, it also weakens the effect of specific patch shape, which is unimportant when considering global structure.

In summary, each scale consists of RGB, depth, normal, and mask channels. These inputs are first encoded independently. Their feature maps are concatenated after the $11$-th convolutional layer, and then pass through the remaining $39$ layers. The local and global scales share weights for the corresponding channels, and their outputs are finally combined with a fully connected layer (Figure 3).

Network Training: The training data for our network are generated from datasets of RGB-D scans of 3D indoor scenes, with high-quality camera poses provided with the datasets. For each RGB-D frame, we segment it into planar patches using agglomerative clustering on the depth channel. For each planar patch, we also estimate its normal based on the depth information. The extracted patches are projected to image space to generate all the necessary channels of input to our network. Very small patches, whose local mask image contains less than $300$ pixels with valid depths, are discarded.

Triplet Focal Loss: When preparing triplets to train our network, we encounter the well-known problem of a severely imbalanced number of negative and positive patch pairs. Given a training sequence, there are many more negative pairs, and most of them are too trivial to help the network learn efficiently. Using randomly sampled triplets would overwhelm the training loss by the easy negatives.

We opt to resolve the imbalance issue by dynamically and discriminatively scaling the losses for hard and easy triplets, inspired by the recent work of focal loss for object detection [44]. Specifically, we propose the triplet focal loss:

[TABLE]

where $x_{a}$, $x_{p}$ and $x_{n}$ are the feature maps extracted for anchor, positive, and negative patches, respectively; $\Delta d_{\text{f}}=d_{\text{f}}(x_{n},x_{a})-d_{\text{f}}(x_{p},x_{a})$, with $d_{\text{f}}$ being the L2 distance between two patch features. Minimizing this loss encouranges the anchor to be closer to the positive patch than to the negative in descriptor space, but with less weight for larger distances.

See Figure 4, left, for a visualization of the loss function with $\alpha=1$. When $\lambda=1$, this loss becomes the usual margined loss, which gives non-negligible loss to easy examples near the margin $\alpha$. When $\lambda>1$, however, we obtain a focal loss that down-weights easy-to-learn triplets while keeping high loss for hard ones. Moreover, it smoothly adjusts the rate at which easy triplets are down-weighted. We found $\lambda=3$ to achieve the best training efficiency (Figure 4, right). Figure 5 shows a t-SNE visualization of coplanarity-based patch features.

3.2 Coplanarity-Based Robust Registration

To investigate the utility of this planar patch descriptor and coplanarity detection approach for 3D reconstruction, we have developed a global registration algorithm that estimates camera poses for an RGB-D video using pairwise constraints derived from coplanar patch matches in addition to keypoint matches.

Our formulation is inspired by the work of Choi et al. [12], where the key feature is the robust penalty term used for automatically selecting the correct matches from a large pool of hypotheses, thus avoiding iterative rematching as in ICP. Note that this formulation does not require an initial alignment of camera poses, which would be required for other SLAM systems that leverage coplanarity constraints.

Given an RGB-D video sequence $\mathcal{F}$, our goal is to compute for each frame $i\in\mathcal{F}$ a camera pose in the global reference frame, $\mathbf{T}_{i}=(\mathbf{R}_{i},\mathbf{t}_{i})$, that brings them into alignment. This is achieved by jointly aligning each pair of frames $(i,j)\in\mathcal{P}$ that were predicted to have some set of coplanar patches, $\Pi_{ij}$. For each pair $\pi=(p,q)\in\Pi_{ij}$, let us suppose w.l.o.g. that patch $p$ is from frame $i$ and $q$ from $j$. Meanwhile, let us suppose a set of matching key-point pairs $\Theta_{ij}$ is detected and matched between frame $i$ and $j$. Similarly, we assume for each point pair $\theta=(\mathbf{u},\mathbf{v})\in\Theta_{ij}$ that key-point $\mathbf{u}$ is from frame $i$ and $\mathbf{v}$ from $j$.

Objective Function: The objective of our coplanarity-based registration contains four terms, responsible for coplanar alignment, coplanar patch pair selection, key-point alignment, and key-point pair selection:

[TABLE]

Given a pair of coplanar patches predicted by the network, the coplanarity data term enforces the coplanarity, via minimizing the point-to-plane distance from sample points on one patch to the plane defined by the other patch:

[TABLE]

where $\delta$ is the coplanarity distance of a patch pair $\pi=(p,q)$. It is computed as the root-mean-square point-to-plane distance over both sets of sample points:

[TABLE]

where $\mathcal{V}_{p}$ is the set of sample points on patch $p$ and $d$ is point-to-plane distance:

[TABLE]

$\phi^{G}_{q}=(\mathbf{p}_{q},\mathbf{n}_{q})$ is the plane defined by patch $q$, which is estimated in the global reference frame using the corresponding transformation $\mathbf{T}_{j}$, and is updated in every iteration. $s_{\pi}$ is a control variable (in $[0,1]$) for the selection of patch pair $\pi$, with $1$ standing for selected and [math] for discarded. $w_{\pi}$ is a weight that measures the confidence of pair $\pi$’s being coplanar. This weight is another connection between the optimization and the network, besides the predicted patch pairs themselves. It is computed based on the feature distance of two patches, denoted by $d_{\text{f}}(p,q)$, extracted by the network: $w_{(p,q)}=e^{-d_{\text{f}}^{2}(p,q)/(\sigma^{2}d_{\text{fm}}^{2})},$ where $d_{\text{fm}}$ is the maximum feature distance and $\sigma=0.6$.

The coplanarity regularization term is defined as:

[TABLE]

where the penalty function is defined as $\Psi(s)=\left(\sqrt{s}-1\right)^{2}$. Intuitively, minimizing this term together with the data term encourages the selection of pairs incurring a small value for the data term, while immediately pruning those pairs whose data term value is too large and deemed to be hard to minimize. $w_{\pi}$ is defined the same as before, and $\mu$ is a weighting variable that controls the emphasis on pair selection.

The key-point data term is defined as:

[TABLE]

Similar to coplanarity, a control variable $s_{\theta}$ is used to determine the selection of point pair $\theta$, subjecting to the key-point regularization term:

[TABLE]

where $\mu$ shares the same weighting variable with Equation (4).

Optimization: The optimization of Equation (7) is conducted iteratively, where each iteration interleaves the optimization of transformations $T$ and selection variables $s$. Ideally, the optimization could take every pair of frames in a sequence as an input for global optimization. However, this is prohibitively expensive since for each frame pair the system scales with the number of patch pairs and key-point pairs. To alleviate this problem, we split the sequence into a list of overlapping fragments, optimize frame poses within each fragment, and then perform a final global registration of the fragments, as in [12].

For each fragment, the optimization takes all frame pairs within that fragment and registers them into a rigid point cloud. After that, we take the matching pairs that have been selected by the intra-fragment optimization, and solve the inter-fragment registration based on those pairs. Inter-fragment registration benefits more from long-range coplanarity predictions.

The putative matches found in this manner are then pruned further with a rapid and approximate RANSAC algorithm applied for each pair of fragments. Given a pair of fragments, we randomly select a set of three matching feature pairs, which could be either planar-patch or key-point pairs. We compute the transformation aligning the selected triplet, and then estimate the “support” for the transformation by counting the number of putative match pairs that are aligned by the transformation. For patch pairs, alignment error is measures by the root-mean-square closest distance between sample points on the two patches. For key-point pairs, we simply use the Euclidean distance. Both use the same threshold of $1$cm. If a transformation is found to be sufficiently supported by the matching pairs (more than $25\%$ consensus), we include all the supporting pairs into the global optimization. Otherwise, we simply discard all putative matches.

Once a set of pairwise constraints have been established in this manner, the frame transformations and pair selection variables are alternately optimized with an iterative process using Ceres [45] for the minimization of the objective function at each iteration. The iterative optimization converges when the relative value change of each unknown is less than $1\times 10^{-6}$. At a convergence, the weighting variable $\mu$, which was initialized to $1$m in the beginning, is decreased by half and the above iterative optimization continues. The whole process is repeated until $\mu$ is lower than $0.01$m, which usually takes less than $50$ iterations. The supplementary material provides a study of the optimization behavior, including convergence and robustness to incorrect pairs.

4 Results and Evaluations

4.1 Training Set, Parameters, and Timings

Our training data is generated from the ScanNet [10] dataset, which contains $1513$ scanned sequences of indoor scenes, reconstructed by BundleFusion [2]. We adopt the training/testing split provided with ScanNet and the training set ($1045$ scenes) are used to generate our training triplets. Each training scene contributes $10K$ triplets. About $10M$ triplets in total are generated from all training scenes. For evaluating our network, we build a coplanarity benchmark using $100$ scenes from the testing set. For hierarchical optimization, the fragment size is $21$, with a $5$-frame overlap between adjacent fragments. The network training takes about $20$ hours to converge. For a sequence of $1K$ frames with $62$ fragments and $30$ patches per frame, the running time is $10$ minutes for coplanarity prediction ($0.1$ second per patch pair) and $20$ minutes for optimization ($5$ minutes for intra-fragment and $15$ minutes for inter-fragment).

4.2 Coplanarity Benchmark

We create a benchmark COP for evaluating RGB-D-based coplanarity matching of planar patches. The benchmark dataset contains $12K$ patch pairs with ground-truth coplanarity, which are organized according to the physical size/area of patches (COP-S) and the centroid distance between pairs of patches (COP-D). COP-S contains $6K$ patch pairs which are split uniformly into three subsets with decreasing average patch size, where the patch pairs are sampled at random distances. COP-D comprises three subsets (each containing $2K$ pairs) with increasing average pair distance but uniformly distributed patch size. For all subsets, the numbers of positive and negative pairs are equal. Details of the benchmark are provided in the supplementary material.

4.3 Network Evaluation

Our network is the first, to our knowledge, that is trained for coplanarity prediction. Therefore, we perform comparison against baselines and ablation studies. See visual results of coplanarity matching in the supplementary material.

Comparing to Baseline Methods: We first compare to two hand-crafted descriptors, namely the color histogram within the patch region and the SIFT feature at the patch centroid. For the task of key-point matching, a commonly practiced method (e.g., in [46]) is to train a neural network that takes image patches centered around the key-points as input. We extend this network to the task of coplanarity prediction, as a non-trivial baseline. For a fair comparison, we train a triplet network with ResNet-50 with only one tower per patch taking three channels (RGB, depth, and normal) as input. For each channel, the image is cropped around the patch centroid, with the same padding and resizing scheme as before. Thus, no mask is needed since the target is always at the image center. We train two networks with different triplets for the task of

1. exact center point matching and
2. coplanarity patch matching, respectively.

The comparison is conducted over COP-S and the results of precision-recall are plotted in Figure 6. The hand-crafted descriptors fail on all tests, which shows the difficulty of our benchmark datasets. Compared to the two alternative center-point-based networks (point matching and coplanarity matching), our method performs significantly better, especially on larger patches.

Ablation Studies: To investigate the need for the various input channels, we compare our full method against that with the RGB, depth, normal, or mask input disabled, over the COP benchmark. To evaluate the effect of multi-scale context, our method is also compared to that without local or global channels. The PR plots in Figure 7 show that our full method works the best for all tests.

From the experiments, several interesting phenomena can be observed. First, the order of overall importance of the different channels is: mask $>$ normal $>$ RGB $>$ depth. This clearly shows that coplanarity prediction across different views can neither rely on appearance or geometry alone. The important role of masking in concentrating the network’s attention is quite evident. We provide a further comparison to justify our specific masking scheme in the supplementary material. Second, the global scale is more effective for bigger patches and more distant pairs, for which the larger scale is required to encode more context. The opposite goes for the local scale due the higher resolution of its input channels. This verifies the complementary effect of the local and global channels in capturing contextual information at different scales.

4.4 Reconstruction Evaluation

Quantitative Results: We perform a quantitative evaluation of reconstruction using the TUM RGB-D dataset by [11], for which ground-truth camera trajectories are available. Reconstruction error is measured by the absolute trajectory error (ATE), i.e., the root-mean-square error (RMSE) of camera positions along a trajectory. We compare our method with six state-of-the-art reconstruction methods, including RGB-D SLAM [47], VoxelHashing [14], ElasticFusion [18], Redwood [12], BundleFusion [2], and Fine-to-Coarse [4]. Note that unlike the other methods, Redwood does not use color information. Fine-to-Coarse is the most closely related to our method, since it uses planar surfaces for structurally-constrained registration. This method, however, relies on a good initialization of camera trajectory to bootstrap, while our method does not. Our method uses SIFT features for key-point detection and matching. We also implement an enhanced version of our method where the key-point matchings are pre-filtered by BundleFusion (named ‘BundleFusion+Ours’).

As an ablation study, we implement five baseline variants of our method.

1. ‘Coplanarity’ is our optimization with only coplanarity constraints. Without key-point matching constraint, our optimization can sometimes be under-determined and needs reformulation to achieve robust registration when not all degrees of freedom (DoFs) can be fixed by coplanarity. The details on the formulation can be found in the supplementary material.
2. ‘Keypoint’ is our optimization with only SIFT key-point matching constraints.
3. ‘No D. in RANSAC’ stands for our method where we did not use our learned patch descriptor during the voting in frame-to-frame RANSAC. In this case, any two patch pairs could cast a vote if they are geometrically aligned by the candidate transformation.
4. ‘No D. in Opt’ means that the optimization objective for coplanarity is not weighted by the matching confidence predicted by our network ($w_{\pi}$ in Equation (3) and (4)).
5. ‘No D. in Both’ is a combination of 3) and 4).

Table 1 reports the ATE RMSE comparison. Our method achieves state-of-the-art results for the first three TUM sequences (the fourth is a flat wall). This is achieved by exploiting our long-range coplanarity matching for robust large-scale loop closure, while utilizing key-point based matching to pin down the possible free DoFs which are not determinable by coplanarity. When being combined with BundleFusion key-points, our method achieves the best results over all sequences. Therefore, our method complements the current state-of-the-art methods by providing a means to handle limited frame-to-frame overlap.

The ablation study demonstrates the importance of our learned patch descriptor in our optimization – i.e., our method performs better than all variants that do not include it. It also shows that coplanarity constraints alone are superior to keypoints only for all sequences except the flat wall (fr3/nst). Using coplanar and keypoint matches together provides the best method overall.

Qualitative Results: Figure 8 shows visual comparisons of reconstruction on sequences from ScanNet [10] and new ones scanned by ourselves. We compare reconstruction results of our method with a state-of-the-art key-point based method (BundleFusion) and a planar-structure-based method (Fine-to-Coarse). The low frame overlap makes the key-point based loop-closure detection fail in BundleFusion. Lost tracking of successive frames provides a poor initial alignment for Fine-to-Coarse, causing it to fail. In contrast, our method can successfully detect non-overlapping loop closures through coplanar patch pairs and achieve good quality reconstructions for these examples without an initial registration. More visual results are shown in the supplementary material.

Effect of Long-Range Coplanarity. To evaluate the effect of long-range coplanarity matching on reconstruction quality, we show in Figure 9 the reconstruction results computed with all, half, and none of the long-range coplanar pairs predicted by our network. We also show a histogram of coplanar pairs survived the optimization. From the visual reconstruction results, the benefit of long-range coplanar pairs is apparent. In particular, the larger scene (bottom) benefits more from long-range coplanarity than the smaller one (top). In Figure 8, we also give the number of non-overlapping coplanar pairs after optimization, showing that long-range coplanarity did help in all examples.

5 Conclusion

We have proposed a new planar patch descriptor designed for finding coplanar patches without a priori global alignment. At its heart, the method uses a deep network to map planar patch inputs with RGB, depth, and normals to a descriptor space where proximity can be used to predict coplanarity. We expect that deep patch coplanarity prediction provides a useful complement to existing features for SLAM applications, especially in scans with large planar surfaces and little inter-frame overlap.

5.0.1 Acknowledgement

We are grateful to Min Liu, Zhan Shi, Lintao Zheng, and Maciej Halber for their help on data preprocessing. We also thank Yizhong Zhang for the early discussions. This work was supported in part by the NSF (VEC 1539014/ 1539099, IIS 1421435, CHS 1617236), NSFC (61532003, 61572507, 61622212), Google, Intel, Pixar, Amazon, and Facebook. Yifei Shi was supported by the China Scholarship Council.

6 Outline

In this supplemental material, we provide the following additional information and results:

• •

Section 7 provides an overview of the dataset of our coplanarity benchmark (COP).

• •

Section 8 gives more evaluation results for our coplanarity network, including a comparison of different masking schemes (Section 8.1), evaluation on patch pairs proposed from real cases of scene reconstruction (Section 8.2), and visual qualitative results of coplanarity matching (Section 8.3).

• •

Section 9 provides more evaluations of the reconstruction algorithm. Specifically, we first evaluate the robustness of the registration against the initial ratio of incorrect pairs (Section 9.1). We then compare the reconstruction performance of two alternative optimization strategies (Section 9.2). Finally, we show more visual results of reconstructions for scenes from various datasets (Section 9.3).

• •

Section 10 discusses the limitations of our method.

• •

Finally, Section 11 provides the formulation for a variatnt of our method that only utilizes coplanarity constraints (Section 11.1), the optimization procedure used for that variant (Section 11.2), and the stability analysis used for achieving a robust optimization in that variant (Section 11.3).

7 COP Benchmark Dataset

Figure 10 and 11 provide an overview of our coplanarity benchmark datasets, COP-S (organized in decreasing patch size) and COP-D (in increasing patch distance), respectively. For each subset, we show both positive and negative pairs, each with two pairs. Note how non-trivial the negative pairs are in our dataset, for example, the negative pairs of S3 and D1.

8 Network Evaluations

This section provides further studies and evaluations of the performance of our coplanarity prediction network.

8.1 Different Masking Schemes

We first investigate several alternative masking schemes for the local and global inputs of our coplanarity network. The proposed masking scheme is summarized as follows (see Figure 12 (right)). The local mask is binary, with the patch of interest in white and the rest of the image in black. The global mask, in contrast, is continuous, with the patch of interest in white and then a smooth decay to black outside the patch boundary.

We compare in Figure 12 our masking scheme (global decay) with several alternatives including

1. using distance-based decaying for both local and global scale (both decay),
2. using distance-based decaying only for local scale (local decay),
3. without decaying for either scale (no decay), and 4) without using a mask at all (no mask). Over the entire COP-D benchmark dataset, we test the above methods and plot the PR curves. The results demonstrate the advantage of our specific design choice of masking scheme (using decaying for global scale but not for local).

8.2 Performance on Patches Proposed during Reconstructions

Our second study investigates the network performance for a realistic balance of positive and negative patch pairs. The performance of our coplanarity network has so far been evaluated over the COP benchmark dataset, which contains comparable numbers of positive and negative examples. To evaluate its performance in a reconstruction setting, we test on patch pairs proposed during the reconstruction of two scenes (the full sequence of ‘fr1/desk’ and ‘fr2/xyz’ from the TUM dataset). The ground-truth coplanarity matching is detected based on the ground-truth alignment provided with the TUM dataset.

Figure 13 shows the plot of PR curves for both intra- and inter-fragment reconstructions. The values for intra-fragment are averaged over all fragments. For patches from the real case of scene reconstruction, our network achieves a precision of $>20\%$, when the recall rate is $80\%$. This accuracy is sufficient for our robust optimization for frame registration, which can be seen from the evaluation in Figure 15; see Section 9.1.

8.3 More Visual Results of Coplanarity Matching

Figure 14 shows some visual results of coplanarity matching. Given a query patch in one frame, we show all patches in another frame, which are color-coded with the dissimilarity predicted by our coplanarity network (blue is small and red is large). The results show that our network produces correct coplanarity embedding, even for patches observed across many views.

9 Reconstruction Evaluations

9.1 Robustness to Initial Coplanarity Accuracy

To evaluate the robustness of our optimization for coplanarity-based alignment, we inspect how tolerant the optimization is to the initial accuracy of the coplanarity prediction. In Figure 15, we plot the reconstruction error of our method on two sequences (full) from TUM dataset, with varying ratio of incorrect input pairs. In our method, given a pair of patches, if their feature distance in the embedding space is smaller than $2.5$, it is used as a hypothetical coplanar pair being input to the optimization. The varying incorrect ratios are thus obtained via gradually introducing more incorrect predictions by adjusting the feature distance threshold.

Reconstruction error is measured by the absolute trajectory error (ATE), i.e., the root-mean-square error (RMSE) of camera positions along a trajectory. The results demonstrate that our method is quite robust against the initial precision of coplanarity matching, for both intra- and inter-fragment reconstructions. In particular, the experiments show that our method is robust for a precision $20\%$ (incorrect ratio of $80\%$), while keeping the recall rate no lower than $80\%$.

9.2 Performance of Different Optimization Strategies

An alternative strategy for solving Equation (2) is to optimize transformations $T$ and selection variables $s$ jointly. In Figure 16, we report the reconstruction error over iterations on two sequences (’fr1/desk’ and ’fr2/xyz’) in the TUM dataset, for both alternating optimization and joint optimization. The reconstruction error is calculated by averaging the ATE of the intra-fragment reconstructions and the inter-fragment reconstructions. The result shows that alternating optimization achieves better convergence performance. Similar results can be observed on other sequences from the TUM dataset.

9.3 More Visual Results of Reconstruction

Figure 17 shows more visual results of reconstruction on $17$ sequences, including $9$ from the ScanNet dataset [10] and $8$ new ones scanned by ourselves. The sequences scanned by ourselves have very sparse loop closure due the missing parts. Our method works well for all these examples. Figure 18 shows the reconstruction of $4$ sequences from the Sun3D dataset [48]. Since the registration of Sun3D sequences is typically shown without fusion in previous works (e.g., [48, 4]), we only show the point clouds. Figure 19 shows two reconstruction examples on scene contains multiple rooms scanned by ourselves.

10 Limitations, Failure Cases and Future work

Our work has several limitations, which suggest topics for future research.

First, coplanarity correspondences alone are not always enough to constrain camera poses uniquely in some environments – e.g., the pose of a camera viewing only a single flat wall will be under-constrained. Therefore, coplanarity is not a replacement for traditional features, such as key-points, lines, etc.; rather, we argue that coplanarity constraints provide additional signal and constraints which are critical in many scanning scenarios, thus helping to improve the reconstruction results. This becomes particularly obvious in scans with a sparse temporal sampling of frames.

Second, for the cases where short-range coplanar patches dominate long-range ones (e.g., a bending wall), our method could reconstruct an overly flat surface due to the coplanarity regularization by false positive coplanar patch pairs between adjacent frames. For example, in Figure 20, we show a tea room scanned by ourselves. The top wall is not flat, but the false positive coplanar pairs detected between adjacent frames could over-regularize the registration, making it mistakenly flattened. This in turn causes the loop cannot be closed at the wall in the bottom.

Third, since the network prediction relies on the information of color, depth and normal, the prediction results could be wrong when the inputs fail to provide sufficient information on coplanarity. For example, two white walls without any context will always be predicted to be coplanar although they could be not.

Last, our optimization is currently a computational bottleneck – it takes approximately $20$ minutes to perform the robust optimization in typical scans shown in the paper. Besides exploiting the highly parallelizable intra-fragment registrations, a more efficient optimization is a worthy direction for future investigation. We are also interested in testing our method on a broader range of RGB-D datasets (e.g. the dataset in [20]).

11 Coplanarity-only Robust Registration

At lines 526-529 of the main paper and in Table 1b, we provide an ablation study in which our method is compared to a variant (called “Coplanarity only”) that uses only predicted matches of coplanar patches to constrain camera poses – i.e., without keypoint matches. In order to produce that one comparative result, we implemented an augmented version of our algorithm that includes a new method for selecting coplanar patch pairs in order to increase the chances of fully constraining the camera pose DoFs with the seletected patch pairs. The following subsections describe that version of the algorithm. Although it is not part of our method usually, we describe it in full here for the sake of reproducibility of the “Coplanarity only” comparison provided in the paper.

11.1 Formulation

Objective Function:

The objective of coplanarity-only registration contains three terms, including the coplanarity data term (Equation (3) of the main paper), the coplanarity regularization term (Equation (4) of the main paper), and a newly introduced frame regularization term for regularizing the optimization based on the assumption that the transformation between adjacent frames is small:

[TABLE]

The frame regularization term $E_{\text{reg-frm}}$ makes sure the system is always solvable, by weakly constraining the transformations of adjacent frames to be as close as possible:

[TABLE]

where $\mathcal{V}_{i}$ is a sparse set of points sampled from frame $i$. $\lambda$ is set to $0.001$ by default.

When using coplanarity constraints only (without key-points), our coplanarity-based alignment may become under-determined or unstable along some DoF, when there are too few coplanar patch pairs that can be used to pin down that DoF. In this case, we must be more willing to keep pairs constraining that DoF, to keep the system stable. To this end, we devise an anisotropic control variable, $\mu$, for patch pair pruning: If some DoF is detected to be unstable and enforcing $p_{k}$ and $q_{k}$ to be coplanar can constrain it, we set $\mu(\pi_{k})$ to be large. The alignment stability is estimated by analyzing the eigenvalues of the 6-DoF alignment error covariance matrix (gradient of the point-to-plane distances w.r.t. the six DoFs in $\mathbf{R}$ and $\mathbf{t}$) as in [49](See details in Section 11.3). Since the stability changes during the optimization, $\mu$ should be updated dynamically, and we describe an optimization scheme with dynamically updated $\mu$ below.

11.2 Optimization

The optimization process is given in Algorithm 1. The core part is solving Equation (7) via alternating optimization of transformations and selection variables (the inner loop in Line 1~1). The iterative process converges when the relative value change of each unknown is less than $1\times 10^{-6}$, which usually takes less than $20$ iterations.

A key step of the optimization is stability analysis and stability-based anisotropic pair pruning (Line 1~1). Since our coplanarity-based alignment is inherently orientation-based, it suffices to inspect the stability of the three translational DoFs. Given a frame $i$, we estimate its translational stability values, denoted by $\gamma^{d}_{i}$ ($d$ is one of the labels of X, Y, and Z-axis), based on the alignment of all frame pairs involving $i$ (see Section 11.3 for details). One can check the stability of frame $i$ along DoF $d$ by examining whether the stability value $\gamma^{d}_{i}$ is greater than a threshold $\gamma_{t}$.

Stability-based anisotropic pair pruning is achieved by dynamically setting the pruning parameter for a patch pair, $\mu(\pi)$ in the coplanarity regularization term (Equation (4) of the main paper). To this end, we set for each frame and each DoF an independent pruning parameter: $\mu^{d}_{i}$ ($i\in\mathcal{F}$ and $d=\text{X},\text{Y},\text{Z}$). They are uniformly set to a relatively large initial value ($0.1$m), and are decreased in each outer loop to gradually allow more pairs to be pruned. For some $\mu^{d}_{i}$, however, if its corresponding stability value $\gamma^{d}_{i}$ is lower than $\gamma_{t}$, it stops decreasing to avoid unstableness. At any given time, the pruning parameter $\mu(\pi)$, with $\pi=(p,q)$, is set to:

[TABLE]

where $d(p)$ is the DoF closest to the normal of patch $p$. The whole process terminates when the stability of all DoFs becomes less than $\gamma_{t}$.

To demonstrate the capability of our optimization to prune incorrect patch pairs, we plot in Figure 21 the ratio of correct coplanarity matches at each iteration step for a ground-truth set. We treat a pair $\pi$ as being kept if its selection variable $s(\pi)>0.5$ and discarded otherwise. With more and more incorrect pairs pruned, the ratio increases while the registration error (measured by absolute camera trajectory error (ATE); see Section 4 of the paper) decreases.

11.3 Stability Analysis

The stability analysis of coplanar alignment is inspired by the work of Gelfand et al. [49] on geometrically stable sampling for point-to-plane ICP. Consider the point-to-plane alignment problem found in the data term of our coplanarity-based registration (see Equation (3) in the main paper). Let us assume we have a collection of points $\mathbf{v}_{p}\in\mathcal{V}_{p}$ sampled from patch $p$, and a plane $\phi_{q}=(\mathbf{p}_{q},\mathbf{n}_{q})$ defined by patch $q$. We want to determine the optimal rotation and translation to be applied to the point set $\mathcal{V}_{p}$, to bring them into coplanar alignment with the plane $\phi_{q}$. In our formulation, source and target patches ($p$ and $q$) are also exchanged to compute alignment error bilaterally (see Line 436 in paper). Below we use only patch $p$ as the source for simplicity of presentation.

We want to minimize the alignment error

[TABLE]

with respect to the rotation $\mathbf{R}$ and translation $\mathbf{t}$.

The rotation is nonlinear, but can be linearized by assuming that incremental rotations will be small:

[TABLE]

for rotations $r_{x}$, $r_{y}$, and $r_{z}$ around the X, Y, and Z axes, respectively. This is equivalent to treating the transformation of $\mathbf{v}_{p}\in\mathcal{V}_{p}$ as a displacement by a vector $[\mathbf{r}\times\mathbf{v}_{p}+\mathbf{t}]$, where $\mathbf{r}=(r_{x},r_{y},r_{z})$. Substituting this into Equation (9), we therefore aim to find a 6-vector $[\mathbf{r}^{T},\mathbf{t}^{T}]$ that minimizes:

[TABLE]

We solve for the aligning transformation by taking partial derivatives of Equation (11) with respect to the transformation parameters in $\mathbf{r}$ and $\mathbf{t}$. This results in a linear system $C\mathbf{x}=\mathbf{b}$ where $\mathbf{x}=[\mathbf{r}^{T},\mathbf{t}^{T}]$ and $\mathbf{b}$ is the residual vector. $C$ is a $6\times 6$ “covariance matrix” of the rotational and translational components, accumulated from the sample points:

[TABLE]

This covariance matrix encodes the increase in the alignment error due to the movement of the transformation parameters from their optimum:

[TABLE]

The larger this increase, the greater the stability of the alignment, since the error landscape will have a deep, well-defined minimum. On the other hand, if there are incremental transformations that cause only a small increase in alignment error, it means the alignment is relatively unstable along that degree of freedom. The analysis of stability can thus be conducted by finding the eigenvalues of matrix $C$. Any small eigenvalues indicate a low-confidence alignment. In our paper, we analyze translational stabilities based on the eigenvalues corresponding to the three translations, $\gamma^{d}$ ($d=\text{X, Y, Z}$).