Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis

TL;DR

This paper introduces BasicTS+, a comprehensive benchmark for fair comparison of multivariate time series forecasting methods, and analyzes heterogeneity in MTS to improve understanding and progress in the field.

Contribution

It presents BasicTS+ for unbiased evaluation of MTS forecasting solutions and classifies MTS based on heterogeneity to guide method selection.

Findings

Benchmarking over 45 solutions reveals current state-of-the-art performance.

Heterogeneity classification aids in selecting suitable forecasting approaches.

Standardized evaluation pipeline improves reproducibility in MTS research.

Abstract

Multivariate Time Series (MTS) analysis is crucial to understanding and managing complex systems, such as traffic and energy systems, and a variety of approaches to MTS forecasting have been proposed recently. However, we often observe inconsistent or seemingly contradictory performance findings across different studies. This hinders our understanding of the merits of different approaches and slows down progress. We address the need for means of assessing MTS forecasting proposals reliably and fairly, in turn enabling better exploitation of MTS as seen in different applications. Specifically, we first propose BasicTS+, a benchmark designed to enable fair, comprehensive, and reproducible comparison of MTS forecasting solutions. BasicTS+ establishes a unified training pipeline and reasonable settings, enabling an unbiased evaluation. Second, we identify the heterogeneity across different MTS as an important consideration and enable classification of MTS based on their temporal and spatial characteristics. Disregarding this heterogeneity is a prime reason for difficulties in selecting the most promising technical directions. Third, we apply BasicTS+ along with rich datasets to assess the capabilities of more than 45 MTS forecasting solutions. This provides readers with an overall picture of the cutting-edge research on MTS forecasting. The code can be accessed at https://github.com/GestaltCogTeam/BasicTS.