Probing Few-Shot Generalization with Attributes

TL;DR

This paper investigates how attribute-based representations affect few-shot learning generalization, revealing that self-supervised pretraining combined with supervised finetuning enhances the ability to generalize to new concepts.

Contribution

It introduces an attribute-based paradigm to analyze concept relatedness and demonstrates the benefits of combining self-supervised pretraining with supervised finetuning for better generalization.

Findings

Supervised learning alone generalizes poorly to new attributes.

Self-supervised pretraining improves generalization to unseen attributes.

Predictability of test attributes estimates model's generalization ability.

Abstract

Despite impressive progress in deep learning, generalizing far beyond the training distribution is an important open challenge. In this work, we consider few-shot classification, and aim to shed light on what makes some novel classes easier to learn than others, and what types of learned representations generalize better. To this end, we define a new paradigm in terms of attributes -- simple building blocks of which concepts are formed -- as a means of quantifying the degree of relatedness of different concepts. Our empirical analysis reveals that supervised learning generalizes poorly to new attributes, but a combination of self-supervised pretraining with supervised finetuning leads to stronger generalization. The benefit of self-supervised pretraining and supervised finetuning is further investigated through controlled experiments using random splits of the attribute space, and we find that predictability of test attributes provides an informative estimate of a model's generalization ability.