Inferring Prototypes for Multi-Label Few-Shot Image Classification with Word Vector Guided Attention

TL;DR

This paper introduces a novel approach for multi-label few-shot image classification that leverages word embeddings to guide attention mechanisms, enabling effective prototype inference for unseen labels without fine-tuning.

Contribution

It proposes a word embedding-guided attention method for prototype inference in ML-FSIC, enhancing generalization to unseen labels without additional training.

Findings

Significant improvement over state-of-the-art on COCO and PASCAL VOC datasets.

Model can infer prototypes for unseen labels without fine-tuning.

Utilizes label embeddings to focus attention on relevant image regions.

Abstract

Multi-label few-shot image classification (ML-FSIC) is the task of assigning descriptive labels to previously unseen images, based on a small number of training examples. A key feature of the multi-label setting is that images often have multiple labels, which typically refer to different regions of the image. When estimating prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data makes this highly challenging. As a solution, in this paper we propose to use word embeddings as a form of prior knowledge about the meaning of the labels. In particular, visual prototypes are obtained by aggregating the local feature maps of the support images, using an attention mechanism that relies on the label embeddings. As an important advantage, our model can infer prototypes for unseen labels without the need for fine-tuning any model parameters, which demonstrates its strong generalization abilities. Experiments on COCO and PASCAL VOC furthermore show that our model substantially improves the current state-of-the-art.