On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings

TL;DR

This paper investigates the 'cone effect' and modality gap in medical vision-language models, introducing a simple control mechanism to optimize cross-modal separation and improve downstream task performance without retraining.

Contribution

It presents a lightweight post-hoc method to control the modality gap in pretrained VLMs, enabling systematic analysis of its impact on medical multimodal tasks.

Findings

Reducing the modality gap improves downstream performance.

Medical datasets are more sensitive to gap modulation.

Complete gap collapse is not always optimal.

Abstract

Vision-Language Models (VLMs) exhibit a characteristic "cone effect" in which nonlinear encoders map embeddings into highly concentrated regions of the representation space, contributing to cross-modal separation known as the modality gap. While this phenomenon has been widely observed, its practical impact on supervised multimodal learning -- particularly in medical domains -- remains unclear. In this work, we introduce a lightweight post-hoc mechanism that keeps pretrained VLM encoders frozen while continuously controlling cross-modal separation through a single hyperparameter {{\lambda}}. This enables systematic analysis of how the modality gap affects downstream multimodal performance without expensive retraining. We evaluate generalist (CLIP, SigLIP) and medically specialized (BioMedCLIP, MedSigLIP) models across diverse medical and natural datasets in a supervised multimodal settings. Results consistently show that reducing excessive modality gap improves downstream performance, with medical datasets exhibiting stronger sensitivity to gap modulation; however, fully collapsing the gap is not always optimal, and intermediate, task-dependent separation yields the best results. These findings position the modality gap as a tunable property of multimodal representations rather than a quantity that should be universally minimized.