Assessing Building Heat Resilience Using UAV and Street-View Imagery with Coupled Global Context Vision Transformer

TL;DR

This paper introduces a machine learning framework combining UAV and street-view imagery with a vision transformer to assess urban building heat resilience, revealing key structural factors linked to heat exposure in Dar es Salaam.

Contribution

It develops a coupled global context vision transformer that fuses UAV and street-view images for scalable heat risk assessment of urban buildings.

Findings

Dual-modality approach outperforms single-modality models by up to 9.3%.

Vegetation, roofing brightness, and material significantly affect heat exposure.

Framework identifies socio-economic inequalities in urban heat risk.

Abstract

Climate change is intensifying human heat exposure, particularly in densely built urban centers of the Global South. Low-cost construction materials and high thermal-mass surfaces further exacerbate this risk. Yet scalable methods for assessing such heat-relevant building attributes remain scarce. We propose a machine learning framework that fuses openly available unmanned aerial vehicle (UAV) and street-view (SV) imagery via a coupled global context vision transformer (CGCViT) to learn heat-relevant representations of urban structures. Thermal infrared (TIR) measurements from HotSat-1 are used to quantify the relationship between building attributes and heat-associated health risks. Our dual-modality cross-view learning approach outperforms the best single-modality models by up to $9.3\%$, demonstrating that UAV and SV imagery provide valuable complementary perspectives on urban structures. The presence of vegetation surrounding buildings (versus no vegetation), brighter roofing (versus darker roofing), and roofing made of concrete, clay, or wood (versus metal or tarpaulin) are all significantly associated with lower HotSat-1 TIR values. Deployed across the city of Dar es Salaam, Tanzania, the proposed framework illustrates how household-level inequalities in heat exposure - often linked to socio-economic disadvantage and reflected in building materials - can be identified and addressed using machine learning. Our results point to the critical role of localized, data-driven risk assessment in shaping climate adaptation strategies that deliver equitable outcomes.