Analysing Multi-Task Regression via Random Matrix Theory with Application to Time Series Forecasting

TL;DR

This paper develops a theoretical framework using random matrix theory for multi-task regression, providing performance estimates and error estimation methods, especially effective in high-dimensional, non-Gaussian data, with applications to time series forecasting.

Contribution

It introduces a novel multi-task regression approach with a closed-form solution and error estimation, linking performance to data and model statistics, advancing hyperparameter tuning and model understanding.

Findings

Improved regression performance on synthetic and real datasets.

Effective error estimation for hyperparameter optimization.

Enhanced multivariate time series forecasting accuracy.

Abstract

In this paper, we introduce a novel theoretical framework for multi-task regression, applying random matrix theory to provide precise performance estimations, under high-dimensional, non-Gaussian data distributions. We formulate a multi-task optimization problem as a regularization technique to enable single-task models to leverage multi-task learning information. We derive a closed-form solution for multi-task optimization in the context of linear models. Our analysis provides valuable insights by linking the multi-task learning performance to various model statistics such as raw data covariances, signal-generating hyperplanes, noise levels, as well as the size and number of datasets. We finally propose a consistent estimation of training and testing errors, thereby offering a robust foundation for hyperparameter optimization in multi-task regression scenarios. Experimental validations on both synthetic and real-world datasets in regression and multivariate time series forecasting demonstrate improvements on univariate models, incorporating our method into the training loss and thus leveraging multivariate information.