Using street view images and visual LLMs to predict heritage values for governance support: Risks, ethics, and policy implications

TL;DR

This paper explores using street view images and multimodal Large Language Models to predict heritage values of buildings in Sweden, aiding policy development while addressing ethical and risk considerations.

Contribution

It introduces a novel application of LLMs for heritage assessment using street view images, providing a scalable method for heritage value identification in policy planning.

Findings

LLMs can predict heritage values with reasonable accuracy.

The approach identified buildings with potential heritage significance across Sweden.

Key risks include transparency issues and error detection challenges.

Abstract

During 2025 and 2026, the Energy Performance of Buildings Directive is being implemented in the European Union member states, requiring all member states to have National Building Renovation Plans. In Sweden, there is a lack of a national register of buildings with heritage values. This is seen as a barrier for the analyses underlying the development of Building Renovation Plans by the involved Swedish authorities. The purpose of this research was to assist Swedish authorities in assigning heritage values to building in the Swedish building stock. As part of the analyses, buildings in street view images from all over Sweden (N=154 710) have been analysed using multimodal Large Language Models (LLM) to assess aspects of heritage value. Zero-shot predictions by LLMs were used as a basis to for identifying buildings with potential heritage values for 5.0 million square meters of heated floor area for the Swedish Building Renovation Plan. In this paper, the results of the predictions and lessons learnt are presented and related to the development of Swedish Building Renovation Plan as part of governance. Potential risks for authorities using LLM-based data are addressed, with a focus on issues of transparency, error detection and sycophancy.