Implementation of remote-sensing models to identify post-disaster health facility damage: Comparative approaches to the 2023 earthquake in Turkey

TL;DR

This paper evaluates AI models using satellite imagery to assess health facility damage after the 2023 Turkey earthquake, finding they can provide rapid reports but lack sufficient accuracy to replace manual assessments.

Contribution

The study introduces a novel method of overlaying AI model outputs with health facility data to estimate damage and identifies key error sources.

Findings

Model A estimated 10.4% damage, while Model B estimated 4.3% across 25 hospitals, 13 dialysis centers, and 454 pharmacies.

Sensitivity was low (42.9%) for both models, but specificity was high (93.6% for Model A, 96.8% for Model B).

Spatially aggregated damage estimates improved sensitivity and agreement compared to individual facility assessments.

Abstract

Earthquakes and other disasters often cause substantial damage to health facilities, impacting short-term response capacity and long-term health system needs. Identifying health facility damage following disasters is therefore crucial for coordinating response, but ground-based evaluations require substantial time and labor. Artificial intelligence (AI) models trained on satellite imagery can estimate building damage and could be used to generate rapid health facility damage reports. There is little published about methods of generating these estimates, testing real-world accuracy, or exploring error. This study presents a novel method of overlaying model damage outputs with health facility location data to generate health facility damage estimates following the February 2023 earthquake in Turkey. Two models were compared for agreement, accuracy, and errors. Building-level damage estimates were obtained for Model A (Microsoft neural network model), and Model B (Google AI model), and overlaid with health facility location data to identify facilities with significant damage. Model agreement, sensitivity and specificity for damage detection were calculated. A descriptive review of common error sources based on selected satellite imagery was conducted. A spatially aggregated damage estimation, based on proportion of buildings damaged in a 0.125km2 area, was also generated and assessed for each model. Twenty-five hospitals, 13 dialysis facilities, and 454 pharmacies were evaluated across three cities. Estimated damage was higher for Model A (10.4%) than Model B (4.3%). Cohen’s kappa was 0.32, indicating fair agreement. Sensitivity was low for both models at 42.9%, while specificity was high (A:93.6%, B:96.8%). Agreement and sensitivity were best for hospitals. Common errors included building identification and underestimation of damage for destroyed buildings. Spatially aggregated damage estimates yielded higher sensitivity (A:71.4%, B:57.1%) and agreement (Cohen’s kappa 0.38). Leveraging remote-sensing models for health facility damage assessment is feasible but currently lacks the sensitivity to replace ground evaluations. Improving building identification, damage detection for destroyed buildings, and spatially aggregating results may improve the performance and utility of these models for use in disaster response settings.

Earthquakes and other disasters can cause significant damage to health facilities. Understanding the scale of impact is important to plan for disaster response efforts and long-term health system rebuilding. Current manual methods of assessing health facility damage, however, can take several weeks to complete. Many research teams have worked to develop artificial intelligence models that use satellite imagery to detect damage to buildings, but there is still limited understanding of how these models perform in real-world settings to identify damage to healthcare facilities. Here, we take two models developed after the February 2023 earthquake in Turkey and overlay their findings with the locations of three types of health facilities: hospitals, dialysis centers, and pharmacies. We examine the accuracy and agreement between the two models and explore sources of error and uncertainty. We found that it was feasible to overlay these data to yield rapid health facility damage reports, but that the sensitivity of the models was low for the health facilities evaluated. We discuss the key sources of error and ways to improve the accuracy and usability of these models for real-world health facility analysis.

Figures