Incorporating Long-term Data in Training Short-term Traffic Prediction Model

TL;DR

This paper investigates how incorporating extensive historical data affects short-term traffic prediction accuracy, identifying challenges like data shifts and proposing methods to improve model performance with large datasets.

Contribution

It is the first study to evaluate the impact of expanding training datasets on traffic prediction accuracy and introduces a novel approach to address data shifts in long-term training.

Findings

Training with 96 months of data can sometimes reduce accuracy due to data disparities.

Proposed weighting scheme effectively manages covariate shift.

Environment aware learning improves model robustness against concept shift.

Abstract

Short-term traffic volume prediction is crucial for intelligent transportation system and there are many researches focusing on this field. However, most of these existing researches concentrated on refining model architecture and ignored amount of training data. Therefore, there remains a noticeable gap in thoroughly exploring the effect of augmented dataset, especially extensive historical data in training. In this research, two datasets containing taxi and bike usage spanning over eight years in New York were used to test such effects. Experiments were conducted to assess the precision of models trained with data in the most recent 12, 24, 48, and 96 months. It was found that the training set encompassing 96 months, at times, resulted in diminished accuracy, which might be owing to disparities between historical traffic patterns and present ones. An analysis was subsequently undertaken to discern potential sources of inconsistent patterns, which may include both covariate shift and concept shift. To address these shifts, we proposed an innovative approach that aligns covariate distributions using a weighting scheme to manage covariate shift, coupled with an environment aware learning method to tackle the concept shift. Experiments based on real word datasets demonstrate the effectiveness of our method which can significantly decrease testing errors and ensure an improvement in accuracy when training with large-scale historical data. As far as we know, this work is the first attempt to assess the impact of contiguously expanding training dataset on the accuracy of traffic prediction models. Besides, our training method is able to be incorporated into most existing short-term traffic prediction models and make them more suitable for long term historical training dataset.