Uncovering large inconsistencies between machine learning derived gridded settlement datasets

TL;DR

This paper assesses the inconsistencies among high-resolution satellite-derived settlement datasets across 42 African countries, revealing significant disagreements and factors influencing these discrepancies, which impact humanitarian and planning efforts.

Contribution

It provides a comprehensive analysis of dataset disagreements and introduces a machine learning model to predict areas of disagreement, highlighting limitations of AI-derived settlement maps.

Findings

Large disagreement between datasets on settlement areas

Geographic and socio-economic factors influence dataset discrepancies

Machine learning can predict disagreement-prone areas

Abstract

High-resolution human settlement maps provide detailed delineations of where people live and are vital for scientific and practical purposes, such as rapid disaster response, allocation of humanitarian resources, and international development. The increased availability of high-resolution satellite imagery, combined with powerful techniques from machine learning and artificial intelligence, has spurred the creation of a wealth of settlement datasets. However, the precise agreement and alignment between these datasets is not known. Here we quantify the overlap of high-resolution settlement map for 42 African countries developed by Google (Open Buildings), Meta (High Resolution Population Maps) and GRID3 (Geo-Referenced Infrastructure and Demographic Data for Development). Across all studied countries we find large disagreement between datasets on how much area is considered settled. We demonstrate that there are considerable geographic and socio-economic factors at play and build a machine learning model to predict for which areas datasets disagree. It it vital to understand the shortcomings of AI derived high-resolution settlement layers as international organizations, governments, and NGOs are already experimenting with incorporating these into programmatic work. As such, we anticipate our work to be a starting point for more critical and detailed analyses of AI derived datasets for humanitarian, planning, policy, and scientific purposes.