Infinite-Task Learning with RKHSs

TL;DR

This paper introduces Infinite Task Learning, a novel approach using operator-valued kernels and RKHSs to learn a continuum of tasks across hyperparameters, with theoretical guarantees and applications in various domains.

Contribution

It extends multi-task learning to an infinite continuum of tasks using RKHSs, providing a new framework with generalization guarantees and practical applications.

Findings

Effective in cost-sensitive classification

Applicable to quantile regression

Supports density level set estimation

Abstract

Machine learning has witnessed tremendous success in solving tasks depending on a single hyperparameter. When considering simultaneously a finite number of tasks, multi-task learning enables one to account for the similarities of the tasks via appropriate regularizers. A step further consists of learning a continuum of tasks for various loss functions. A promising approach, called \emph{Parametric Task Learning}, has paved the way in the continuum setting for affine models and piecewise-linear loss functions. In this work, we introduce a novel approach called \emph{Infinite Task Learning} whose goal is to learn a function whose output is a function over the hyperparameter space. We leverage tools from operator-valued kernels and the associated vector-valued RKHSs that provide an explicit control over the role of the hyperparameters, and also allows us to consider new type of constraints. We provide generalization guarantees to the suggested scheme and illustrate its efficiency in cost-sensitive classification, quantile regression and density level set estimation.