FILM: How can Few-Shot Image Classification Benefit from Pre-Trained Language Models?

TL;DR

This paper introduces a novel few-shot image classification framework leveraging pre-trained language models and contrastive learning, with a specially designed textual branch and metric module optimized via MAML, achieving improved transferability and performance.

Contribution

It presents a new framework that fully utilizes semantic information from pre-trained language models for few-shot learning, surpassing existing methods that only enhance standard modules.

Findings

Effective alignment of visual and textual features achieved

Improved few-shot classification accuracy on multiple benchmarks

Enhanced transferability through MAML-based training

Abstract

Few-shot learning aims to train models that can be generalized to novel classes with only a few samples. Recently, a line of works are proposed to enhance few-shot learning with accessible semantic information from class names. However, these works focus on improving existing modules such as visual prototypes and feature extractors of the standard few-shot learning framework. This limits the full potential use of semantic information. In this paper, we propose a novel few-shot learning framework that uses pre-trained language models based on contrastive learning. To address the challenge of alignment between visual features and textual embeddings obtained from text-based pre-trained language model, we carefully design the textual branch of our framework and introduce a metric module to generalize the cosine similarity. For better transferability, we let the metric module adapt to different few-shot tasks and adopt MAML to train the model via bi-level optimization. Moreover, we conduct extensive experiments on multiple benchmarks to demonstrate the effectiveness of our method.