Bayesian learning of feature spaces for multitasks problems

TL;DR

This paper presents a Bayesian framework that unifies Kernel Machines and Extreme Learning Machines for multitask regression, optimizing kernel parameters and promoting task sharing, leading to improved performance.

Contribution

It introduces RFF-BLR, a novel Bayesian model connecting KMs and ELMs via RFFs, enabling joint optimization and multi-task knowledge transfer.

Findings

Significant performance improvements over state-of-the-art methods

Effective kernel parameter and model complexity optimization

Successful multi-task knowledge transfer demonstrated

Abstract

This paper introduces a novel approach for multi-task regression that connects Kernel Machines (KMs) and Extreme Learning Machines (ELMs) through the exploitation of the Random Fourier Features (RFFs) approximation of the RBF kernel. In this sense, one of the contributions of this paper shows that for the proposed models, the KM and the ELM formulations can be regarded as two sides of the same coin. These proposed models, termed RFF-BLR, stand on a Bayesian framework that simultaneously addresses two main design goals. On the one hand, it fits multitask regressors based on KMs endowed with RBF kernels. On the other hand, it enables the introduction of a common-across-tasks prior that promotes multioutput sparsity in the ELM view. This Bayesian approach facilitates the simultaneous consideration of both the KM and ELM perspectives enabling (i) the optimisation of the RBF kernel parameter $\gamma$ within a probabilistic framework, (ii) the optimisation of the model complexity, and (iii) an efficient transfer of knowledge across tasks. The experimental results show that this framework can lead to significant performance improvements compared to the state-of-the-art methods in multitask nonlinear regression.