Learning to Learn with Contrastive Meta-Objective

TL;DR

This paper introduces ConML, a contrastive meta-objective that leverages task identity as additional supervision in meta-learning, improving generalization and performance across various models with minimal extra cost.

Contribution

It proposes a novel contrastive meta-objective that enhances meta-learning by utilizing task identity, applicable across different meta-learners and in-context learning models.

Findings

ConML significantly boosts meta-learning performance.

It integrates seamlessly with existing meta-learners.

The approach requires minimal additional implementation effort.

Abstract

Meta-learning enables learning systems to adapt quickly to new tasks, similar to humans. Different meta-learning approaches all work under/with the mini-batch episodic training framework. Such framework naturally gives the information about task identity, which can serve as additional supervision for meta-training to improve generalizability. We propose to exploit task identity as additional supervision in meta-training, inspired by the alignment and discrimination ability which is is intrinsic in human's fast learning. This is achieved by contrasting what meta-learners learn, i.e., model representations. The proposed ConML is evaluating and optimizing the contrastive meta-objective under a problem- and learner-agnostic meta-training framework. We demonstrate that ConML integrates seamlessly with existing meta-learners, as well as in-context learning models, and brings significant boost in performance with small implementation cost.