MTL2L: A Context Aware Neural Optimiser

TL;DR

This paper introduces MTL2L, a context-aware neural optimizer that adapts its update rules based on input data, enabling effective learning across unseen input domains during meta-testing.

Contribution

The paper presents MTL2L, a novel neural optimizer that incorporates multi-task learning to handle input-domain heterogeneity in meta-learning.

Findings

MTL2L successfully adapts to unseen input domains.

It outperforms previous neural optimizers on heterogeneous datasets.

MTL2L demonstrates improved generalization in meta-learning scenarios.

Abstract

Learning to learn (L2L) trains a meta-learner to assist the learning of a task-specific base learner. Previously, it was shown that a meta-learner could learn the direct rules to update learner parameters; and that the learnt neural optimiser updated learners more rapidly than handcrafted gradient-descent methods. However, we demonstrate that previous neural optimisers were limited to update learners on one designated dataset. In order to address input-domain heterogeneity, we introduce Multi-Task Learning to Learn (MTL2L), a context aware neural optimiser which self-modifies its optimisation rules based on input data. We show that MTL2L is capable of updating learners to classify on data of an unseen input-domain at the meta-testing phase.