Representation as a Service

TL;DR

This paper proposes a novel method for machine learning service providers to leverage previous task knowledge, creating representations that minimize generalization error for new tasks, thus enabling rapid, accurate learning in diverse scenarios.

Contribution

It introduces a new approach that minimizes an empirical proxy of intra-task generalization error, improving task transfer and lifelong learning performance.

Findings

Achieves state-of-the-art results in single-task transfer

Demonstrates effectiveness in multitask learning

Excels in lifelong learning scenarios

Abstract

Consider a Machine Learning Service Provider (MLSP) designed to rapidly create highly accurate learners for a never-ending stream of new tasks. The challenge is to produce task-specific learners that can be trained from few labeled samples, even if tasks are not uniquely identified, and the number of tasks and input dimensionality are large. In this paper, we argue that the MLSP should exploit knowledge from previous tasks to build a good representation of the environment it is in, and more precisely, that useful representations for such a service are ones that minimize generalization error for a new hypothesis trained on a new task. We formalize this intuition with a novel method that minimizes an empirical proxy of the intra-task small-sample generalization error. We present several empirical results showing state-of-the art performance on single-task transfer, multitask learning, and the full lifelong learning problem.