ARGENT: Adaptive Hierarchical Image-Text Representations

TL;DR

This paper introduces ARGENT, a hyperbolic vision-language model that effectively captures hierarchical structures in image-text representations, improving state-of-the-art performance and proposing new evaluation metrics for hierarchy understanding.

Contribution

The paper presents an adaptive hyperbolic embedding method with a novel entailment loss and a new hierarchical evaluation protocol, advancing hyperbolic VLM capabilities.

Findings

ARGENT outperforms previous hyperbolic VLMs on multiple benchmarks.

The adaptive loss prevents cone collapse and maintains hierarchy integrity.

New hierarchical metrics provide more reliable evaluation of hierarchical understanding.

Abstract

Large-scale Vision-Language Models (VLMs) such as CLIP learn powerful semantic representations but operate in Euclidean space, which fails to capture the inherent hierarchical structure of visual and linguistic concepts. Hyperbolic geometry, with its exponential volume growth, offers a principled alternative for embedding such hierarchies with low distortion. However, existing hyperbolic VLMs use entailment losses that are unstable: as parent embeddings contract toward the origin, their entailment cones widen toward a half-space, causing catastrophic cone collapse that destroys the intended hierarchy. Additionally, hierarchical evaluation of these models remains unreliable, being largely retrieval-based and correlation-based metrics and prone to taxonomy dependence and ambiguous negatives. To address these limitations, we propose an adaptive entailment loss paired with a norm regularizer that prevents cone collapse without heuristic aperture clipping. We further introduce an angle-based probabilistic entailment protocol (PEP) for evaluating hierarchical understanding, scored with AUC-ROC and Average Precision. This paper introduces a stronger hyperbolic VLM baseline ARGENT, Adaptive hieRarchical imaGe-tExt represeNTation. ARGENT improves the SOTA hyperbolic VLM by 0.7, 1.1, and 0.8 absolute points on image classification, text-to-image retrieval, and proposed hierarchical metrics, respectively.