Cross-Modal Few-Shot Learning with Second-Order Neural Ordinary Differential Equations

TL;DR

SONO introduces a second-order neural ODE approach for cross-modal few-shot learning, improving generalization and reducing overfitting by leveraging text-based augmentation and efficient initialization, outperforming existing methods.

Contribution

The paper proposes SONO, a novel second-order neural ODE framework that enhances cross-modal few-shot learning with improved expressiveness and data augmentation techniques.

Findings

Outperforms state-of-the-art in few-shot learning tasks

Utilizes text-based augmentation with CLIP for data enrichment

Reduces overfitting through second-order neural ODEs

Abstract

We introduce SONO, a novel method leveraging Second-Order Neural Ordinary Differential Equations (Second-Order NODEs) to enhance cross-modal few-shot learning. By employing a simple yet effective architecture consisting of a Second-Order NODEs model paired with a cross-modal classifier, SONO addresses the significant challenge of overfitting, which is common in few-shot scenarios due to limited training examples. Our second-order approach can approximate a broader class of functions, enhancing the model's expressive power and feature generalization capabilities. We initialize our cross-modal classifier with text embeddings derived from class-relevant prompts, streamlining training efficiency by avoiding the need for frequent text encoder processing. Additionally, we utilize text-based image augmentation, exploiting CLIP's robust image-text correlation to enrich training data significantly. Extensive experiments across multiple datasets demonstrate that SONO outperforms existing state-of-the-art methods in few-shot learning performance.