AutoML Meets Time Series Regression Design and Analysis of the AutoSeries Challenge

TL;DR

This paper presents the design, analysis, and results of the AutoSeries challenge, an automated machine learning competition for time series regression across diverse datasets, emphasizing minimal human intervention and continuous model adaptation.

Contribution

It introduces the first AutoML challenge focused on time series regression, showcasing effective automated solutions and analyzing their performance across multiple real-world datasets.

Findings

Participants improved performance significantly over baseline

Feature engineering combined with LightGBM was highly effective

Additional time did not significantly improve results

Abstract

Analyzing better time series with limited human effort is of interest to academia and industry. Driven by business scenarios, we organized the first Automated Time Series Regression challenge (AutoSeries) for the WSDM Cup 2020. We present its design, analysis, and post-hoc experiments. The code submission requirement precluded participants from any manual intervention, testing automated machine learning capabilities of solutions, across many datasets, under hardware and time limitations. We prepared 10 datasets from diverse application domains (sales, power consumption, air quality, traffic, and parking), featuring missing data, mixed continuous and categorical variables, and various sampling rates. Each dataset was split into a training and a test sequence (which was streamed, allowing models to continuously adapt). The setting of time series regression, differs from classical forecasting in that covariates at the present time are known. Great strides were made by participants to tackle this AutoSeries problem, as demonstrated by the jump in performance from the sample submission, and post-hoc comparisons with AutoGluon. Simple yet effective methods were used, based on feature engineering, LightGBM, and random search hyper-parameter tuning, addressing all aspects of the challenge. Our post-hoc analyses revealed that providing additional time did not yield significant improvements. The winners' code was open-sourced https://github.com/NehzUx/AutoSeries.