Like Humans to Few-Shot Learning through Knowledge Permeation of Vision and Text

TL;DR

This paper introduces BiKop, a bidirectional knowledge permeation strategy that enhances few-shot learning by integrating visual and textual information to better recognize novel classes.

Contribution

The paper proposes a novel hierarchical joint representation method with bidirectional knowledge permeation for improved few-shot learning performance.

Findings

BiKop outperforms existing methods on four benchmarks.

Disentangling base-class semantics improves novel class recognition.

Bidirectional knowledge flow enhances generalization in few-shot tasks.

Abstract

Few-shot learning aims to generalize the recognizer from seen categories to an entirely novel scenario. With only a few support samples, several advanced methods initially introduce class names as prior knowledge for identifying novel classes. However, obstacles still impede achieving a comprehensive understanding of how to harness the mutual advantages of visual and textual knowledge. In this paper, we propose a coherent Bidirectional Knowledge Permeation strategy called BiKop, which is grounded in a human intuition: A class name description offers a general representation, whereas an image captures the specificity of individuals. BiKop primarily establishes a hierarchical joint general-specific representation through bidirectional knowledge permeation. On the other hand, considering the bias of joint representation towards the base set, we disentangle base-class-relevant semantics during training, thereby alleviating the suppression of potential novel-class-relevant information. Experiments on four challenging benchmarks demonstrate the remarkable superiority of BiKop. Our code will be publicly available.