Hyperdimensional Cross-Modal Alignment of Frozen Language and Image Models for Efficient Image Captioning

TL;DR

This paper introduces HDFLIM, a novel framework that aligns frozen vision and language models in a shared hyperdimensional space using symbolic operations, enabling efficient image captioning without model fine-tuning.

Contribution

HDFLIM demonstrates that cross-modal alignment can be achieved through symbolic hyperdimensional operations on frozen models, avoiding costly retraining and fine-tuning.

Findings

HDFLIM achieves comparable performance to end-to-end training methods.

Generated captions are more semantically grounded than zero-shot baselines.

Alignment is accomplished without modifying pretrained models.

Abstract

Large unimodal foundation models for vision and language encode rich semantic structures, yet aligning them typically requires computationally intensive multimodal fine-tuning. Such approaches depend on large-scale parameter updates, are resource intensive, and can perturb pretrained representations. Emerging evidence suggests, however, that independently trained foundation models may already exhibit latent semantic compatibility, reflecting shared structures in the data they model. This raises a fundamental question: can cross-modal alignment be achieved without modifying the models themselves? Here we introduce HDFLIM (HyperDimensional computing with Frozen Language and Image Models), a framework that establishes cross-modal mappings while keeping pretrained vision and language models fully frozen. HDFLIM projects unimodal embeddings into a shared hyperdimensional space and leverages lightweight symbolic operations -- binding, bundling, and similarity-based retrieval to construct associative cross-modal representations in a single pass over the data. Caption generation emerges from high-dimensional memory retrieval rather than iterative gradient-based optimization. We show that HDFLIM achieves performance comparable to end-to-end vision-language training methods and produces captions that are more semantically grounded than zero-shot baselines. By decoupling alignment from parameter tuning, our results suggest that semantic mapping across foundation models can be realized through symbolic operations on hyperdimensional encodings of the respective embeddings. More broadly, this work points toward an alternative paradigm for foundation model alignment in which frozen models are integrated through structured representational mappings rather than through large-scale retraining. The codebase for our implementation can be found at https://github.com/Abhishek-Dalvi410/HDFLIM.