Efficacy of regularized multi-task learning based on SVM models

TL;DR

This paper analyzes the theoretical properties and practical performance of a regularized multi-task learning framework based on SVMs, showing it is reliable with large data and benefits mainly in early learning stages.

Contribution

It provides the first theoretical analysis of M-SVM's consistency, task interaction behavior, and convergence rates, clarifying when and how MTL improves learning.

Findings

M-SVM is Bayes risk consistent with large data.

Task interaction diminishes as data size increases.

M-SVM improves pre-convergence-rate factors, especially with small data.

Abstract

This paper investigates the efficacy of a regularized multi-task learning (MTL) framework based on SVM (M-SVM) to answer whether MTL always provides reliable results and how MTL outperforms independent learning. We first find that M-SVM is Bayes risk consistent in the limit of large sample size. This implies that despite the task dissimilarities, M-SVM always produces a reliable decision rule for each task in terms of misclassification error when the data size is large enough. Furthermore, we find that the task-interaction vanishes as the data size goes to infinity, and the convergence rates of M-SVM and its single-task counterpart have the same upper bound. The former suggests that M-SVM cannot improve the limit classifier's performance; based on the latter, we conjecture that the optimal convergence rate is not improved when the task number is fixed. As a novel insight of MTL, our theoretical and experimental results achieved an excellent agreement that the benefit of the MTL methods lies in the improvement of the pre-convergence-rate factor (PCR, to be denoted in Section III) rather than the convergence rate. Moreover, this improvement of PCR factors is more significant when the data size is small.