Spatially explicit accuracy assessment of deep learning-based, fine-resolution built-up land data in the United States

TL;DR

This study evaluates the accuracy of a deep learning-based global built-up land dataset in the US, translating probabilistic outputs into binary maps and introducing a novel visual assessment tool, revealing high accuracy levels.

Contribution

It introduces a method to binarize probabilistic built-up data and a new visual tool for spatial accuracy assessment, improving understanding of deep learning-based land cover data.

Findings

Optimal threshold of 0.5 maximizes agreement with reference data.

High county-level F-1 scores (~0.8) indicate strong accuracy.

Consistent accuracy across rural-urban gradients.

Abstract

Geospatial datasets derived from remote sensing data by means of machine learning methods are often based on probabilistic outputs of abstract nature, which are difficult to translate into interpretable measures. For example, the Global Human Settlement Layer GHS-BUILT-S2 product reports the probability of the presence of built-up areas in 2018 in a global 10 m x 10 m grid. However, practitioners typically require interpretable measures such as binary surfaces indicating the presence or absence of built-up areas or estimates of sub-pixel built-up surface fractions. Herein, we assess the relationship between the built-up probability in GHS-BUILT-S2 and reference built-up surface fractions derived from a highly reliable reference database for several regions in the United States. Furthermore, we identify a binarization threshold using an agreement maximization method that creates binary built-up land data from these built-up probabilities. These binary surfaces are input to a spatially explicit, scale-sensitive accuracy assessment which includes the use of a novel, visual-analytical tool which we call focal precision-recall signature plots. Our analysis reveals that a threshold of 0.5 applied to GHS-BUILT-S2 maximizes the agreement with binarized built-up land data derived from the reference built-up area fraction. We find high levels of accuracy (i.e., county-level F-1 scores of almost 0.8 on average) in the derived built-up areas, and consistently high accuracy along the rural-urban gradient in our study area. These results reveal considerable accuracy improvements in human settlement models based on Sentinel-2 data and deep learning, in both rural and urban areas, as compared to earlier, Landsat-based versions of the Global Human Settlement Layer.