HMID-Net: An Exploration of Masked Image Modeling and Knowledge Distillation in Hyperbolic Space

TL;DR

HMID-Net introduces a novel approach combining Masked Image Modeling and knowledge distillation in hyperbolic space, effectively capturing hierarchical visual-semantic structures and outperforming existing models in various tasks.

Contribution

This work is the first to integrate MIM and knowledge distillation within hyperbolic space for efficient hierarchical modeling.

Findings

Achieves comparable success to Euclidean methods in hyperbolic space

Significantly outperforms MERU and CLIP in classification and retrieval

Demonstrates the effectiveness of hyperbolic space techniques in visual-semantic tasks

Abstract

Visual and semantic concepts are often structured in a hierarchical manner. For instance, textual concept `cat' entails all images of cats. A recent study, MERU, successfully adapts multimodal learning techniques from Euclidean space to hyperbolic space, effectively capturing the visual-semantic hierarchy. However, a critical question remains: how can we more efficiently train a model to capture and leverage this hierarchy? In this paper, we propose the Hyperbolic Masked Image and Distillation Network (HMID-Net), a novel and efficient method that integrates Masked Image Modeling (MIM) and knowledge distillation techniques within hyperbolic space. To the best of our knowledge, this is the first approach to leverage MIM and knowledge distillation in hyperbolic space to train highly efficient models. In addition, we introduce a distillation loss function specifically designed to facilitate effective knowledge transfer in hyperbolic space. Our experiments demonstrate that MIM and knowledge distillation techniques in hyperbolic space can achieve the same remarkable success as in Euclidean space. Extensive evaluations show that our method excels across a wide range of downstream tasks, significantly outperforming existing models like MERU and CLIP in both image classification and retrieval.