Global-Local Airborne Mapping (GLAM): Reconstructing a City from Aerial Videos

TL;DR

This paper introduces a scalable, real-time visual SLAM system for aerial videos that can reconstruct large city-scale maps with high accuracy using tens of thousands of frames on a standard computer.

Contribution

The authors present a novel feature-based aerial SLAM system capable of large-area, city-scale mapping with near real-time performance and high accuracy, scalable to tens of thousands of frames.

Findings

Reconstructed city-scale maps with 2-meter accuracy.

Achieved near real-time processing of 90,000 frames.

Demonstrated the largest and fastest aerial map reconstruction to date.

Abstract

Monocular visual SLAM has become an attractive practical approach for robot localization and 3D environment mapping, since cameras are small, lightweight, inexpensive, and produce high-rate, high-resolution data streams. Although numerous robust tools have been developed, most existing systems are designed to operate in terrestrial environments and at relatively small scale (a few thousand frames) due to constraints on computation and storage. In this paper, we present a feature-based visual SLAM system for aerial video whose simple design permits near real-time operation, and whose scalability permits large-area mapping using tens of thousands of frames, all on a single conventional computer. Our approach consists of two parallel threads: the first incrementally creates small locally consistent submaps and estimates camera poses at video rate; the second aligns these submaps with one another to produce a single globally consistent map via factor graph optimization over both poses and landmarks. Scale drift is minimized through the use of 7-degree-of-freedom similarity transformations during submap alignment. We quantify our system's performance on both simulated and real data sets, and demonstrate city-scale map reconstruction accurate to within 2 meters using nearly 90,000 aerial video frames - to our knowledge, the largest and fastest such reconstruction to date.

Tables

Table 1. Table 1: Results on synthetic datasets. Comparison of reconstructed maps to ground truth demonstrates errors on the order of centimeters and millimeters. The number of points in synth3 was made intentionally smaller than those in synth1 and synth2. Figures of the reconstruction from the three synthetic datasets can be found in the supplmentary material.

	Frames	Submaps	Points	RMSE(cm)
synth1	1100	14	239727	0.3
synth2	3408	15	328337	3.4
synth3	40000	53	505000	0.6

Table 2. Table 2: Comparison with Visual SFM. ICP results on the first 5k frames of Downtown datset. GLAM is much faster than Visual SFM with comparable accuracy. Note that the LIDAR points are quantized to one meter spacing.

	RMSE	Runtime
Visual SFM	2.88 m	3.96 hrs
GLAM	2.93 m	11.79 min

Equations

$$f(S,L)=\sum _{i\in S}\sum _{j\in L}||s_i{R}_i{x}_{ij}+t_i-X_j|{|}^2+\sum _{i\in S}||{\lambda }_a{\mu }_i|{|}^2$$

$$G_{up}$$

$$\upsilon$$

$$G$$

$$\frac{\partial y}{\partial R}=-y_{\times },\frac{\partial y}{\partial t}=I,\frac{\partial y}{\partial s}=y$$

$$y=(y_1,y_2,y_3)=f(S,L)$$

$$y_{\times }=\left[\begin{array}{ccc}0& -y_3& y_2\\ y_3& 0& -y_1\\ -y_2& y1& 0\end{array}\right]$$

Full text

In this paper, we present a feature-based visual SLAM system for aerial video whose simple design permits near real-time operation, and whose scalability permits large-area mapping using tens of thousands of frames, all on a single conventional computer. Our approach consists of two parallel threads: the first incrementally creates small locally consistent submaps and estimates camera poses at video rate; the second aligns these submaps with one another to produce a single globally consistent map via factor graph optimization over both poses and landmarks. Scale drift is minimized through the use of 7-degree-of-freedom similarity transformations during submap alignment.

We quantify our system’s performance on both simulated and real data sets, and demonstrate city-scale map reconstruction accurate to within 2 meters using nearly $90,000$ aerial video frames - to our knowledge, the largest and fastest such reconstruction to date.

††*Work performed while at Vision Systems Inc.

1 Introduction

Recent progress in unmanned aerial vehicle (UAV) technology has led to exponential growth in the use of drones with onboard sensing. Originally designed for military applications, camera-equipped UAVs are now commonplace in domains such as commercial surveillance, photography, disaster management, product delivery, and mapping [6]. An essential step toward better utilization of aerial videos and autonomous drones is real-time localization of the vehicle and accurate mapping of the observed world. Localization relying solely on onboard inertial and GPS measurements, however, cannot achieve pixel-level accuracy due to error accumulation, latency, and relatively coarse precision; furthermore, onboard sensing can be unreliable in GPS-denied environments. Visual simultaneous localization and mapping (SLAM) attempts to address these challenges using camera images to augment or replace other sensors.

Our proposed method, nicknamed Global-Local Airborne Mapping (GLAM), approaches large-scale visual SLAM by partitioning the camera’s video stream into small local submaps created using point features, epipolar geometry between keyframes, point triangulation, and incremental bundle adjustment (BA). Submaps are aligned globally using a graph-based least squares optimization that minimizes the distance between corresponding 3D points. Small temporal overlap ensures that sequential submaps observe common scene content and hence provides correspondences for alignment, while restricting submap size to a small number of frames mitigates drift accumulation. Associations between non-overlapping submaps, for instance due to loop closures, are detected via fast bag-of-visual-words recognition. This paper demonstrates that such a system can run in near real-time on videos of the scale of a hundred thousand frames.

2 Related Work

Monocular SLAM is a long-researched subject that has seen evolution from filter based [26, 7] to keyframe based [25, 22] approaches. PTAM [12], originally developed for augmented reality applications, was one of the first widely used and practical real-time SLAM systems, but was limitated in terms of scale and robustness. Subsequent research improved upon this point feature and bundle adjustment methods [24, 22]. More sophisticated monocular SLAM approaches like DTAM [19] and LSD-SLAM [8] perform semi-dense reconstruction by optimizing directly on image intensities rather than discrete features. SVO [9] is a semi-direct approach monocular SLAM approach designed specifically for Micro Aerial Vehicles (MAVs); although SVO operates at very high frame rates, it does not perform loop closure and has been evaluated only on small datasets. ORB-SLAM [17] is a recent feature-based approach that has shown good performance on a wide variety of datasets, but largely untested for large aerial datasets.

City-scale 3D reconstruction has received some attention in the structure-from-motion research community [18]. Pollefeys et al. [21] reconstruct parts of a city from a hundred thousand frames using INS and metadata to simplify the computation. Agarwal et al. [3] and Heinly et al. [11] perform city and world scale reconstruction respectively from large collections of photographs utilizing cloud computing. Google Earth provides 3D models of a few selected cities, but the technology behind its reconstructions is largely unknown. Leotta et al.[14] have published open-source software for aerial SfM focused on videos of a scale of a thousand frames but do not address scaling beyond that.

Real-time reconstruction of a city from aerial videos using purely visual data has remained largely unexplored in the literature: traditional visual SLAM approaches do not address the challenges of very long image sequences and large-scale maps, while structure-from-motion methods require substantial offline computation and/or information from additional sensors. Our system incorporates elements of both visual SLAM and large-scale 3D reconstruction, demonstrating fast and accurate city-scale mapping on aerial videos nearly 90,000 frames in length on a single consumer grade computer.

3 Proposed Approach

Like many existing real-time SLAM systems, we adopt a multi-threaded strategy for computational efficiency, partitioning the problem into tasks that run in parallel. However, rather than defining the tasks as tracking and mapping [12, 9, 17], we define tasks as local submap creation and global submap alignment. Figure 1 shows an overview of GLAM’s building blocks.

The submap creation thread operates similarly to previous point features and keyframe based approaches [12, 17, 24] that extract point features from video frames, determine relative pose, triangulate points, and perform incremental bundle adjustment. Building the Hessian matrix for incremental bundle adjustment is the most computationally expensive step [23]: as the number of frames processed increases, the cost of building the Hessian at every BA iteration becomes prohibitive for real-time operation. We therefore limit each submap to a small number of keyframes in order to maintain a relatively consistent and bounded processing rate.

Each submap consists of a set of 3D landmarks. The submap alignment thread creates and optimizes a pose-graph to determine the set of 7-DoF similarity transformations, one per submap, that minimizes the distances between corresponding landmarks. In contrast to existing approaches that close loops by creating a global database of keyframes (e.g.  [17]), our approach closes loops by building a visual vocabulary of submaps.

GLAM has been evaluated on both synthetic and real datasets. The former are created by simulating drone flight trajectories over an aerial LIDAR point cloud [1], while the latter consists of a large continuous aerial video (labeled “Downtown”) with nearly $90,000$ 1-megapixel frames captured at $30$ fps.

The main contribution of this paper is a novel visual SLAM pipeline that (1) partitions work into parallel threads of fast local submap creation and large-scale global submap alignment; (2) has the ability to close large loops (3) reconstructs accurate city-size maps from aerial videos; and (4) operates in near real-time at the scale of a hundred thousand images.

3.1 Local Submap Creation

The following sections describe keyframe-based submap creation in greater detail. The central idea of keyframe-based SLAM is to use only frames that have sufficiently distinct information for 3D reconstruction. Each video frame is processed in sequence, with keyframes created periodically to reduce redundancy and improve efficiency—only the keyframes are used in intra-submap bundle adjustment.

Feature Extraction. Point features are extracted from each video frame. We use SIFT features [16] throughout our system due to their proven robustness to viewpoint, scale and orientation changes. SIFT extraction and matching can be slow, so we use a GPU implementation [27] for greater efficiency.

Tracking. Existing 3D landmarks are projected into the current frame using the pose of the previous frame, since inter-frame motion is assumed to be small. Points that fall outside the frame or that were originally viewed from a substantially different angle are removed. The SIFT descriptors of the remaining points are then matched to those of the current image features to obtain initial 2D-3D correspondences. The 3D pose of the current frame is then estimated using Perspective-n-Point (PnP) localization [15] embedded in a RANSAC outer loop.

PnP can fail if the video is discontinuous or if the viewpoint has changed drastically between frames. If the number of PnP inliers is too low, the submap is terminated at the previous frame and a new submap is initiated at the current frame.

Keyframes. As new parts of the world come into view, the number of PnP inliers continually decreases. When this number falls below a threshold $\tau_{resection}$, the system adds a new keyframe and triangulates new 3D points. Selection of $\tau_{resection}$ is key in ensuring a balance between speed and stability: a high value causes frequent keyframe additions and thus slows the overall system, while a low value reduces visual overlap between keyframes and may lead to unstable or failed bundle adjustment.

To improve stability further, an additional keyframe is chosen halfway between the previous keyframe and the current frame. This middle keyframe’s pose is initialized to its PnP estimate, and its 2D PnP inliers are added as observations of their counterpart 3D landmarks for BA. Geometrically consistent 2D matches are obtained between the middle keyframe and the current frame via RANSAC-based essential matrix estimation, and these matches are triangulated using the PnP pose estimates to form new 3D landmarks and corresponding 2D observations for BA. The SIFT descriptor from the middle keyframe is assigned to each landmark.

Since the triangulation of new landmarks does not take into account the landmarks that have already been constructed, a subset may be duplicated. Therefore, any new landmark whose descriptor matches any existing landmark is removed. The current frame is finally added as a keyframe and its pose and observations are added to BA.

This selection strategy depends on neither physical nor temporal distance between keyframes; instead, keyframes are added only when sufficient new visual information is available, allowing the system to process videos acquired at arbitrary speed. In the Downtown dataset, for instance, the system selected keyframes spaced roughly 25 to 100 frames apart.

Incremental Bundle Adjustment. Similar to PTAM [12] and ORB-SLAM [17], GLAM runs BA every time a new keyframe is added, optimizing the poses of all keyframes and the positions of all landmarks while holding fixed the intrinsic camera parameters. Different libraries including g2o [13] and ceres-solver [4] were evaluated, but pba [28] performed best due to its specialization for BA problems.

Bootstrapping. When a new submap is initiated, no keyframes or 3D landmarks yet exist, so a strategy different from the steady-state process described above must be employed to establish initial geometry. The first frame of the submap becomes the first keyframe, with its pose fixed at the origin. Features from subsequent frames are matched against those of the first keyframe to find epipolar geometry via RANSAC-based essential matrix estimation [10]. The current frame becomes the second keyframe if the number of RANSAC inliers falls below $\tau_{stereo}$ or if the average triangulation angle exceeds $\alpha_{stereo}$. (for the experiments described in this paper, we used $\tau_{stereo}=1000$ and $\alpha_{stereo}=30^{\circ}$). The inlier correspondences are triangulated to form the initial 3D landmarks.

Completion. A submap is sucessfully completed when the number of keyframes exceeds a threshold. We found that 20 keyframes were generally sufficient to form a stable reconstruction and minimize internal loop closures. Upon completion, all frames are re-localized to the final landmark set via PnP, which requires only that 3D-2D matches and not features to be recomputed. The next submap is initiated to have a degree of overlap (in our experiments, 10%) with the current submap so that the two share a set of 3D points for alignment.

Outlier filtering. Incorrectly matched point features lead to outliers that can dramatically affect bundle adjustment accuracy. Our system therefore filters outliers at several stages. First, after every bundle adjustment step, any point whose reprojection error exceeds a threshold is removed. Some incorrect matches may exhibit small reprojection error, such as when triangulation rays are nearly parallel and the reconstructed landmark is very far from the rest of the scene. To remove these points, the k (30) nearest neighbours of each point are computed using a kd-tree, and a point is removed if its average distance from its neighbours is more than 2 standard deviations. Finally, upon submap completion, any point with large reprojection error in all frames is removed.

Focal length estimation. Use of accurate camera intrinsics is instrumental in full-scale reconstruction, since there is no single rectifying transformation for incorrect intrinsics. The published focal length of 1800 for the lens used to collect the Downtown dataset was imprecise and led to reconstruction failures. We therefore included per-keyframe focal length in PBA optimization, observing that the median over all frames’ estimated focal length converged to 1751 as the number of frames increased, as shown in Figure 3.

3.2 Global Submap Alignment

As local submaps are created, a parallel thread uses them as the unit of processing for full map reconstruction. This thread discovers correspondences between submaps, including loop closures, and optimizes pose and structure globally and efficiently.

Submap matching. Although temporally adjacent submaps are known to be associated with one another by construction, it is required to discover visual links between submaps that view common scene content, regardless of the time at which they were created. This forms a connectivity graph in which submaps constitute nodes and commonly observed points constitute edges.

A naive implementation would identify visual links through brute-force search over every possible pair of submaps. This requires $O(n^{2})$ time for $n$ submaps, making the process computationally intractable as $n$ grows. To form a connectivity graph efficiently, we employ a bag-of-words technique. A vocabulary tree is created offline using randomly-sampled SIFT descriptors drawn from aerial images; a series of k-means clustering operations partition the descriptors into a tree of visual words [20]. As each submap is completed, its landmarks’ SIFT descriptors are added to a database formed over the tree, and visual words are incrementally associated with the submap via an inverted index. The set of landmark descriptors in this submap is then used to query the database, which returns a ranked set of matching submaps and their weighted match scores. This mechanism reduces the complexity of submap matching from quadratic to linear time, as shown in Figure 6.

The result is a graph whose edges represent potential visual matches based on an unordered bag of indexed visual words with no geometric constraints. Some graph edges may be incorrect, sharing a number of similar-looking features but not actually viewing common areas of the scene (e.g.  due to repeated urban structures such as windows). The geometric consistency of correspondences is verified by estimating a closed form similarity transform between 3D landmarks in a RANSAC loop, accounting for both pose and scale differences. Edges with too few inliers are removed, as are all outlier landmark correspondences.

Submap pose-graph After establishing temporal and spatial correspondences between submaps, the next step is to create a pose-graph and optimize the pose of the submaps such that the 3D distance between corresponding points is minimized (Figure 4). Scale-drift [24] within submaps is addressed implicitly by the keyframe addition strategy, while scale-drift across submaps is adressed by using a 7 DoF similarity transform to represent each submap’s pose.

The distance between corresponding 3D points of the submaps is minimized by using non-linear Gauss-Newton (GN) optimization [10], which involves finding a Jacobian w.r.t. to all free parameters. In this case, the 7 DoF pose of a submap and the $\mathbb{R}^{3}$ location of a landmark are the free parameters of the cost function. Differentiating the cost function w.r.t. the Lie group elements (R,s,t) yields the Lie algebra space sim(3) [2]. sim(3) is a 7 dimensional vector space ($\omega$, $\sigma$, $\mu$), where $\omega$, $\sigma$ and $\mu$ respresent the Lie algebra components of rotation, translation and scale respectively.

Given a set of submaps $S$ and landmarks $L$, a 7 DoF similarity transform $(R,s,t)$ is associated with every submap $i$ and represented in the pose-graph with a Sim(3) node. A world point $X_{j}$ is represented with a $\mathbb{R}^{3}$ node in the pose-graph. If submap $i$ observes 3D point $X_{j}$, it is represented by edge $x_{ij}$ in the pose-graph. Scale is constrained by multiplication of a small constant $\lambda_{a}$ ($<$1) with the Lie algebra of scale, $\mu_{i}$. The scale prior prevents the solver from converging to a trivial solution, i.e zero scale. The resulting cost function is:

[TABLE]

Initially, the scale prior used was $\sum_{i\in S}||c-s_{i}||^{2}$, where c was a small constant, typically 1. When added to the remaining cost function, this scale prior prevented the trivial solution by evaluating to a non-zero value. However, it is an invalid operation since it is subtracting the Lie group $s_{i}$ from 1, and addition / subtraction operations are undefined for Lie groups. Hence, a valid scale prior instead constraints the scale by multiplying the Lie Algebra of scale $\mu$ with a small constant $\lambda$. This scale prior has a similar effect on the overall cost function at $\mu$ equals zero. It was also observed experimentally that a cost function with the former scale prior resulted in submaps that didn’t align properly to each other but the latter scale prior fixed those alignment issues.

Every step of a GN optimization updates the sim(3) vector of each submap vertex with a $\upsilon_{up}$ vector. If $\upsilon$ and $G$ are the Lie algebra and Lie group of the current state of a Sim(3) vertex, then the vector update $\upsilon_{up}$ is applied as follows:

[TABLE]

The g2o [13] framework is used to form and optimize the factor graph via Levenberg-Marquardt (LM), with robustness added via Huber loss [10]. Preconditioned conjugate gradient is used to efficiently solve the sparse linear system of equations arising at each step of LM, and analytic Jacobians were dervied to further speed up the computation. and are as follows:

[TABLE]

where,

[TABLE]

4 Experiments

We evaluated GLAM on both synthetic and real datasets. Our experiments demonstrate centimeter level accuracy on synthetic datasets, and accuracy comparable to VisualSFM [29] on real datasets, while providing a much faster run-time.

4.1 Synthetic Data

We created synthetic datasets using Google Earth to simulate camera trajectories and using a LIDAR point cloud [1] to serve as ground truth for simulated image projections. The results of 3D reconstruction on three such datasets is shown in Figure 5 and summarized in Table 1.

4.2 Real Data

The Downtown dataset (Figure 2) is a video of 88,100 frames shot over Providence, USA. We applied GLAM to the entire video; Figure 6 illustrates runtime performance and scale, and Figure 7 visualizes the reconstruction results. The average run-time for submap creation is 5.8 fps, while the latency of the submap alignment and reconstruction thread is typically a few seconds.

We registered the landmark cloud reconstructed in the first 5,000 frames to a LIDAR point cloud of the same area using ICP [5] to quantify accuracy with respect to ground truth. We also compared performance with VisualSFM [29], run on every $10^{th}$ frame of the first 5,000 frames (Table 2).

5 Conclusion

This paper introduces a new visual SLAM system that can process long videos in near real-time on a single computer. Local submaps are constructed using incremental bundle adjustment over keyframes, and the global scene is reconstructed using factor graph optimization over submaps rather than keyframes, which allows closure of large loops. Submap creation has constant runtime complexity throughout, while submap alignment time increases linearly at a low rate, indicating that the system can be run on even larger datasets while maintaining acceptable latency.

The system’s main performance bottleneck is currently SIFT feature extraction and matching. Future work will focus on improving run time by using binary descriptors. We also plan to investigate reducing the total number of constraints, incorporation of global averaging techniques, testing the approach on even larger datasets, and applying the system to terrestrial videos.

6 Acknowledgements

The authors would like to thank Vishal Jain his helpful advice and guidance on different parts of the project. This research is based upon work partly supported by Air Force Research Laboratory(AFRL) under contract number FA8650-14-C-1826. This document is approved for public release via 88ABW-2017-2724.