Lightweight and Interpretable Transformer via Mixed Graph Algorithm Unrolling for Traffic Forecast

TL;DR

This paper introduces a lightweight, interpretable transformer-like neural network for traffic forecasting, unrolled from a mixed-graph optimization algorithm, achieving competitive accuracy with fewer parameters.

Contribution

It proposes a novel unrolled graph-based optimization approach to build an interpretable transformer model for spatio-temporal traffic prediction.

Findings

Achieves competitive accuracy with state-of-the-art methods.

Reduces model parameter count significantly.

Incorporates graph learning modules as self-attention mechanisms.

Abstract

Unlike conventional "black-box" transformers with classical self-attention mechanism, we build a lightweight and interpretable transformer-like neural net by unrolling a mixed-graph-based optimization algorithm to forecast traffic with spatial and temporal dimensions. We construct two graphs: an undirected graph $\mathcal{G}^u$ capturing spatial correlations across geography, and a directed graph $\mathcal{G}^d$ capturing sequential relationships over time. We predict future samples of signal $\mathbf{x}$, assuming it is "smooth" with respect to both $\mathcal{G}^u$ and $\mathcal{G}^d$, where we design new $\ell_2$ and $\ell_1$-norm variational terms to quantify and promote signal smoothness (low-frequency reconstruction) on a directed graph. We design an iterative algorithm based on alternating direction method of multipliers (ADMM), and unroll it into a feed-forward network for data-driven parameter learning. We periodically insert graph learning modules for $\mathcal{G}^u$ and $\mathcal{G}^d$ that play the role of self-attention. Experiments show that our unrolled networks achieve competitive traffic forecast performance as state-of-the-art prediction schemes, while reducing parameter counts drastically.