Revisiting Fine-tuning for Few-shot Learning

TL;DR

This paper revisits fine-tuning for few-shot learning, demonstrating its competitive performance on various datasets and analyzing techniques to improve stability and accuracy during the process.

Contribution

It shows that simple fine-tuning can outperform or match advanced few-shot algorithms across multiple tasks and provides insights into optimizing the fine-tuning process.

Findings

Fine-tuning achieves higher accuracy than many few-shot algorithms on mini-ImageNet.

Using low learning rates stabilizes the retraining process.

Adaptive optimizers and full-network updates improve test accuracy.

Abstract

Few-shot learning is the process of learning novel classes using only a few examples and it remains a challenging task in machine learning. Many sophisticated few-shot learning algorithms have been proposed based on the notion that networks can easily overfit to novel examples if they are simply fine-tuned using only a few examples. In this study, we show that in the commonly used low-resolution mini-ImageNet dataset, the fine-tuning method achieves higher accuracy than common few-shot learning algorithms in the 1-shot task and nearly the same accuracy as that of the state-of-the-art algorithm in the 5-shot task. We then evaluate our method with more practical tasks, namely the high-resolution single-domain and cross-domain tasks. With both tasks, we show that our method achieves higher accuracy than common few-shot learning algorithms. We further analyze the experimental results and show that: 1) the retraining process can be stabilized by employing a low learning rate, 2) using adaptive gradient optimizers during fine-tuning can increase test accuracy, and 3) test accuracy can be improved by updating the entire network when a large domain-shift exists between base and novel classes.