Training-Free Dual Hyperbolic Adapters for Better Cross-Modal Reasoning

TL;DR

This paper introduces Training-free Dual Hyperbolic Adapters (T-DHA), a novel method that embeds hierarchical semantic relationships in hyperbolic space to enhance cross-modal reasoning in vision-language models without additional training.

Contribution

The paper proposes a training-free hyperbolic adapter that models hierarchical semantic data in hyperbolic space, improving efficiency and performance in cross-modal reasoning tasks.

Findings

T-DHA outperforms state-of-the-art methods in few-shot image recognition.

T-DHA demonstrates superior domain generalization capabilities.

Hyperbolic embedding enhances hierarchical data representation.

Abstract

Recent research in Vision-Language Models (VLMs) has significantly advanced our capabilities in cross-modal reasoning. However, existing methods suffer from performance degradation with domain changes or require substantial computational resources for fine-tuning in new domains. To address this issue, we develop a new adaptation method for large vision-language models, called \textit{Training-free Dual Hyperbolic Adapters} (T-DHA). We characterize the vision-language relationship between semantic concepts, which typically has a hierarchical tree structure, in the hyperbolic space instead of the traditional Euclidean space. Hyperbolic spaces exhibit exponential volume growth with radius, unlike the polynomial growth in Euclidean space. We find that this unique property is particularly effective for embedding hierarchical data structures using the Poincar\'e ball model, achieving significantly improved representation and discrimination power. Coupled with negative learning, it provides more accurate and robust classifications with fewer feature dimensions. Our extensive experimental results on various datasets demonstrate that the T-DHA method significantly outperforms existing state-of-the-art methods in few-shot image recognition and domain generalization tasks.