A Statistical Model for Predicting Generalization in Few-Shot Classification

TL;DR

This paper introduces a Gaussian feature distribution model to predict generalization error in few-shot classification, addressing the lack of validation data and improving estimation accuracy over existing methods.

Contribution

It presents a novel Gaussian model for feature distributions and an unbiased distance estimator to predict generalization error in few-shot learning scenarios.

Findings

Outperforms leave-one-out cross-validation in accuracy

Provides reliable generalization error estimates with limited samples

Enhances understanding of feature distribution impacts on classification

Abstract

The estimation of the generalization error of classifiers often relies on a validation set. Such a set is hardly available in few-shot learning scenarios, a highly disregarded shortcoming in the field. In these scenarios, it is common to rely on features extracted from pre-trained neural networks combined with distance-based classifiers such as nearest class mean. In this work, we introduce a Gaussian model of the feature distribution. By estimating the parameters of this model, we are able to predict the generalization error on new classification tasks with few samples. We observe that accurate distance estimates between class-conditional densities are the key to accurate estimates of the generalization performance. Therefore, we propose an unbiased estimator for these distances and integrate it in our numerical analysis. We empirically show that our approach outperforms alternatives such as the leave-one-out cross-validation strategy.