BOIL: Towards Representation Change for Few-shot Learning

TL;DR

This paper introduces BOIL, a meta-learning algorithm that emphasizes representation change over reuse by updating only the model's extractor during inner loop, leading to improved cross-domain few-shot learning performance.

Contribution

BOIL is a novel meta-learning method that updates only the feature extractor, demonstrating the importance of representation change for effective few-shot learning across domains.

Findings

BOIL outperforms MAML on cross-domain tasks.

Representation change is crucial for domain-agnostic few-shot learning.

Empirical analysis shows feature vectors move quickly to their target representations.

Abstract

Model Agnostic Meta-Learning (MAML) is one of the most representative of gradient-based meta-learning algorithms. MAML learns new tasks with a few data samples using inner updates from a meta-initialization point and learns the meta-initialization parameters with outer updates. It has recently been hypothesized that representation reuse, which makes little change in efficient representations, is the dominant factor in the performance of the meta-initialized model through MAML in contrast to representation change, which causes a significant change in representations. In this study, we investigate the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks. To this aim, we propose a novel meta-learning algorithm, called BOIL (Body Only update in Inner Loop), which updates only the body (extractor) of the model and freezes the head (classifier) during inner loop updates. BOIL leverages representation change rather than representation reuse. This is because feature vectors (representations) have to move quickly to their corresponding frozen head vectors. We visualize this property using cosine similarity, CKA, and empirical results without the head. BOIL empirically shows significant performance improvement over MAML, particularly on cross-domain tasks. The results imply that representation change in gradient-based meta-learning approaches is a critical component.