On the Role of Neural Collapse in Meta Learning Models for Few-shot Learning

TL;DR

This paper investigates neural collapse phenomena in meta-learning models for few-shot learning, revealing partial collapse behaviors and their dependence on model size, which enhances understanding of learned representations in such models.

Contribution

First to analyze neural collapse in meta-learning frameworks for few-shot learning, providing insights into feature behaviors and model properties during training.

Findings

Features tend to exhibit neural collapse with increased model size

Complete neural collapse is not fully achieved in meta-learning models

Neural collapse properties are partially observed in few-shot learning scenarios

Abstract

Meta-learning frameworks for few-shot learning aims to learn models that can learn new skills or adapt to new environments rapidly with a few training examples. This has led to the generalizability of the developed model towards new classes with just a few labelled samples. However these networks are seen as black-box models and understanding the representations learnt under different learning scenarios is crucial. Neural collapse ($\mathcal{NC}$) is a recently discovered phenomenon which showcases unique properties at the network proceeds towards zero loss. The input features collapse to their respective class means, the class means form a Simplex equiangular tight frame (ETF) where the class means are maximally distant and linearly separable, and the classifier acts as a simple nearest neighbor classifier. While these phenomena have been observed in simple classification networks, this study is the first to explore and understand the properties of neural collapse in meta learning frameworks for few-shot learning. We perform studies on the Omniglot dataset in the few-shot setting and study the neural collapse phenomenon. We observe that the learnt features indeed have the trend of neural collapse, especially as model size grows, but to do not necessarily showcase the complete collapse as measured by the $\mathcal{NC}$ properties.