Online Multi-Task Learning with Recursive Least Squares and Recursive Kernel Methods

TL;DR

This paper presents two innovative online multi-task learning methods based on recursive least squares and kernel techniques, offering exact recursive solutions with competitive computational costs and improved wind speed forecasting accuracy.

Contribution

Introduces two novel recursive online multi-task learning algorithms using graph-based formulations and task-stacking, with structural task relationship modeling and efficient recursive updates.

Findings

Both methods outperform existing online MTL approaches in wind speed forecasting.

The proposed algorithms achieve quadratic per-instance computational cost.

Significant performance improvements demonstrated in real-world case study.

Abstract

This paper introduces two novel approaches for Online Multi-Task Learning (MTL) Regression Problems. We employ a high performance graph-based MTL formulation and develop two alternative recursive versions based on the Weighted Recursive Least Squares (WRLS) and the Online Sparse Least Squares Support Vector Regression (OSLSSVR) strategies. Adopting task-stacking transformations, we demonstrate the existence of a single matrix incorporating the relationship of multiple tasks and providing structural information to be embodied by the MT-WRLS method in its initialization procedure and by the MT-OSLSSVR in its multi-task kernel function. Contrasting the existing literature, which is mostly based on Online Gradient Descent (OGD) or cubic inexact approaches, we achieve exact and approximate recursions with quadratic per-instance cost on the dimension of the input space (MT-WRLS) or on the size of the dictionary of instances (MT-OSLSSVR). We compare our online MTL methods to other contenders in a real-world wind speed forecasting case study, evidencing the significant gain in performance of both proposed approaches.