MAML is a Noisy Contrastive Learner in Classification

TL;DR

This paper reinterprets MAML as a noisy contrastive learner, revealing its contrastive nature and identifying interference issues, then proposes a simple technique to improve its performance, validated through extensive experiments.

Contribution

It offers a new perspective on MAML as a contrastive learning process and introduces the zeroing trick to reduce interference, enhancing its effectiveness.

Findings

MAML is analogous to a supervised contrastive learner.

The zeroing trick reduces interference from random initialization.

Proposed method improves performance on mini-ImageNet and Omniglot.

Abstract

Model-agnostic meta-learning (MAML) is one of the most popular and widely adopted meta-learning algorithms, achieving remarkable success in various learning problems. Yet, with the unique design of nested inner-loop and outer-loop updates, which govern the task-specific and meta-model-centric learning, respectively, the underlying learning objective of MAML remains implicit and thus impedes a more straightforward understanding of it. In this paper, we provide a new perspective of the working mechanism of MAML. We discover that MAML is analogous to a meta-learner using a supervised contrastive objective. The query features are pulled towards the support features of the same class and against those of different classes. Such contrastiveness is experimentally verified via an analysis based on the cosine similarity. Moreover, we reveal that vanilla MAML has an undesirable interference term originating from the random initialization and the cross-task interaction. We thus propose a simple but effective technique, zeroing trick, to alleviate the interference. Extensive experiments are conducted on both mini-ImageNet and Omniglot datasets to demonstrate the consistent improvement brought by our proposed method, validating its effectiveness.