Hybrid hidden Markov LSTM for short-term traffic flow prediction

TL;DR

This paper introduces a hybrid hidden Markov-LSTM model for short-term traffic flow prediction, combining the strengths of both models to better capture complex traffic dynamics and improve prediction accuracy.

Contribution

The paper proposes a novel hybrid HMM-LSTM architecture that leverages the advantages of both models for enhanced traffic flow prediction performance.

Findings

Significant performance improvements over traditional Markov switching ARIMA.

Hybrid model captures complex dynamic patterns more effectively.

Outperforms standalone LSTM in traffic prediction tasks.

Abstract

Deep learning (DL) methods have outperformed parametric models such as historical average, ARIMA and variants in predicting traffic variables into short and near-short future, that are critical for traffic management. Specifically, recurrent neural network (RNN) and its variants (e.g. long short-term memory) are designed to retain long-term temporal correlations and therefore are suitable for modeling sequences. However, multi-regime models assume the traffic system to evolve through multiple states (say, free-flow, congestion in traffic) with distinct characteristics, and hence, separate models are trained to characterize the traffic dynamics within each regime. For instance, Markov-switching models with a hidden Markov model (HMM) for regime identification is capable of capturing complex dynamic patterns and non-stationarity. Interestingly, both HMM and LSTM can be used for modeling an observation sequence from a set of latent or, hidden state variables. In LSTM, the latent variable is computed in a deterministic manner from the current observation and the previous latent variable, while, in HMM, the set of latent variables is a Markov chain. Inspired by research in natural language processing, a hybrid hidden Markov-LSTM model that is capable of learning complementary features in traffic data is proposed for traffic flow prediction. Results indicate significant performance gains in using hybrid architecture compared to conventional methods such as Markov switching ARIMA and LSTM.