MuseCL: Predicting Urban Socioeconomic Indicators via Multi-Semantic Contrastive Learning

TL;DR

MuseCL introduces a multi-semantic contrastive learning framework that effectively combines visual and textual data to improve the prediction of urban socioeconomic indicators, demonstrating significant performance gains across multiple cities.

Contribution

This work presents a novel multi-semantic contrastive learning approach that jointly models visual and textual data for urban socioeconomic prediction, with an innovative cross-modality fusion mechanism.

Findings

MuseCL achieves an average of 10% improvement in R^2 over baselines.

Contrastive sample pairs enhance semantic feature extraction from images.

Cross-modality attention improves the integration of visual and textual features.

Abstract

Predicting socioeconomic indicators within urban regions is crucial for fostering inclusivity, resilience, and sustainability in cities and human settlements. While pioneering studies have attempted to leverage multi-modal data for socioeconomic prediction, jointly exploring their underlying semantics remains a significant challenge. To address the gap, this paper introduces a Multi-Semantic Contrastive Learning (MuseCL) framework for fine-grained urban region profiling and socioeconomic prediction. Within this framework, we initiate the process by constructing contrastive sample pairs for street view and remote sensing images, capitalizing on the similarities in human mobility and Point of Interest (POI) distribution to derive semantic features from the visual modality. Additionally, we extract semantic insights from POI texts embedded within these regions, employing a pre-trained text encoder. To merge the acquired visual and textual features, we devise an innovative cross-modality-based attentional fusion module, which leverages a contrastive mechanism for integration. Experimental results across multiple cities and indicators consistently highlight the superiority of MuseCL, demonstrating an average improvement of 10% in $R^2$ compared to various competitive baseline models. The code of this work is publicly available at https://github.com/XixianYong/MuseCL.