MTP: Exploring Multimodal Urban Traffic Profiling with Modality Augmentation and Spectrum Fusion

TL;DR

This paper introduces MTP, a multimodal framework for urban traffic profiling that leverages frequency domain analysis and spectrum fusion of numeric, visual, and textual data to improve traffic signal understanding and prediction.

Contribution

The paper presents a novel multimodal traffic profiling method that integrates frequency-based visual, textual, and numeric data with hierarchical contrastive learning, enhancing traffic signal analysis.

Findings

Outperforms state-of-the-art methods on six real-world datasets.

Effectively fuses multimodal spectrum information for better traffic prediction.

Demonstrates the importance of multimodal and frequency domain analysis in urban traffic modeling.

Abstract

With rapid urbanization in the modern era, traffic signals from various sensors have been playing a significant role in monitoring the states of cities, which provides a strong foundation in ensuring safe travel, reducing traffic congestion and optimizing urban mobility. Most existing methods for traffic signal modeling often rely on the original data modality, i.e., numerical direct readings from the sensors in cities. However, this unimodal approach overlooks the semantic information existing in multimodal heterogeneous urban data in different perspectives, which hinders a comprehensive understanding of traffic signals and limits the accurate prediction of complex traffic dynamics. To address this problem, we propose a novel Multimodal framework, MTP, for urban Traffic Profiling, which learns multimodal features through numeric, visual, and textual perspectives. The three branches drive for a multimodal perspective of urban traffic signal learning in the frequency domain, while the frequency learning strategies delicately refine the information for extraction. Specifically, we first conduct the visual augmentation for the traffic signals, which transforms the original modality into frequency images and periodicity images for visual learning. Also, we augment descriptive texts for the traffic signals based on the specific topic, background information and item description for textual learning. To complement the numeric information, we utilize frequency multilayer perceptrons for learning on the original modality. We design a hierarchical contrastive learning on the three branches to fuse the spectrum of three modalities. Finally, extensive experiments on six real-world datasets demonstrate superior performance compared with the state-of-the-art approaches.