NODE-Adapter: Neural Ordinary Differential Equations for Better Vision-Language Reasoning

TL;DR

This paper introduces NODE-Adapter, a novel approach using Neural Ordinary Differential Equations to enhance vision-language reasoning, addressing resource demands, parameter efficiency, and modality integration for improved downstream task adaptation.

Contribution

The paper proposes NODE-Adapter, which models prototype optimization as a continuous process with Neural ODEs, improving vision-language reasoning and task adaptation.

Findings

Outperforms state-of-the-art methods in few-shot classification

Effective in domain generalization tasks

Enhances visual reasoning in human-object interaction scenarios

Abstract

In this paper, we consider the problem of prototype-based vision-language reasoning problem. We observe that existing methods encounter three major challenges: 1) escalating resource demands and prolonging training times, 2) contending with excessive learnable parameters, and 3) fine-tuning based only on a single modality. These challenges will hinder their capability to adapt Vision-Language Models (VLMs) to downstream tasks. Motivated by this critical observation, we propose a novel method called NODE-Adapter, which utilizes Neural Ordinary Differential Equations for better vision-language reasoning. To fully leverage both visual and textual modalities and estimate class prototypes more effectively and accurately, we divide our method into two stages: cross-modal prototype construction and cross-modal prototype optimization using neural ordinary differential equations. Specifically, we exploit VLM to encode hand-crafted prompts into textual features and few-shot support images into visual features. Then, we estimate the textual prototype and visual prototype by averaging the textual features and visual features, respectively, and adaptively combine the textual prototype and visual prototype to construct the cross-modal prototype. To alleviate the prototype bias, we then model the prototype optimization process as an initial value problem with Neural ODEs to estimate the continuous gradient flow. Our extensive experimental results, which cover few-shot classification, domain generalization, and visual reasoning on human-object interaction, demonstrate that the proposed method significantly outperforms existing state-of-the-art approaches.