Parallel Non-divergent Flow Accumulation For Trillion Cell Digital Elevation Models On Desktops Or Clusters

TL;DR

This paper introduces a parallel flow accumulation algorithm for digital elevation models that guarantees fixed memory access per cell, enabling efficient processing of trillion-cell datasets on desktops or clusters with superior scalability and resource use.

Contribution

The paper presents a novel parallel algorithm for flow accumulation that ensures fixed memory access per cell, scalable to trillion-cell datasets, outperforming previous methods.

Findings

Ran on datasets up to two trillion cells

Achieved ~30% scaling efficiency up to 48 cores

Completed processing of the largest dataset in 24 minutes

Abstract

Continent-scale datasets challenge hydrological algorithms for processing digital elevation models. Flow accumulation is an important input for many such algorithms; here, I parallelize its calculation. The new algorithm works on one or many cores, or multiple machines, and can take advantage of large memories or cope with small ones. Unlike previous parallel algorithms, the new algorithm guarantees a fixed number of memory access and communication events per raster cell. In testing, the new algorithm ran faster and used fewer resources than previous algorithms, exhibiting ~30% strong and weak scaling efficiencies up to 48 cores and linear scaling across datasets ranging over three orders of magnitude. The largest dataset tested had two trillion (2*10^12) cells. With 48 cores, processing required 24 minutes wall-time (14.5 compute-hours). This test is three orders of magnitude larger than any previously performed in the literature. Complete, well-commented source code and correctness tests are available on Github.