Learning-to-Learn Stochastic Gradient Descent with Biased Regularization

TL;DR

This paper introduces a meta-learning approach for stochastic gradient descent that learns a bias vector to improve performance across tasks, with theoretical guarantees and practical efficiency.

Contribution

It proposes a novel meta-algorithm for estimating and updating a bias vector in SGD, enhancing learning-to-learn capabilities with theoretical bounds and practical efficiency.

Findings

Bias-aware SGD outperforms unbiased SGD when task variance is low.

The meta-algorithm efficiently updates the bias with low space and time complexity.

Numerical experiments confirm the effectiveness of the proposed approach.

Abstract

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.