Visual Lexicon: Rich Image Features in Language Space

TL;DR

Visual Lexicon introduces a novel visual language encoding rich image details into text space, enabling high-fidelity image reconstruction and improved vision-language tasks without fine-tuning, bridging visual and linguistic representations.

Contribution

It presents ViLex, a self-supervised method to encode detailed visual information into language tokens, enhancing image reconstruction and vision-language model performance.

Findings

Achieves higher fidelity in image reconstruction with a single token.

Enables zero-shot DreamBooth tasks without fine-tuning.

Improves performance across 15 vision-language benchmarks.

Abstract

We present Visual Lexicon, a novel visual language that encodes rich image information into the text space of vocabulary tokens while retaining intricate visual details that are often challenging to convey in natural language. Unlike traditional methods that prioritize either high-level semantics (e.g., CLIP) or pixel-level reconstruction (e.g., VAE), ViLex simultaneously captures rich semantic content and fine visual details, enabling high-quality image generation and comprehensive visual scene understanding. Through a self-supervised learning pipeline, ViLex generates tokens optimized for reconstructing input images using a frozen text-to-image (T2I) diffusion model, preserving the detailed information necessary for high-fidelity semantic-level reconstruction. As an image embedding in the language space, ViLex tokens leverage the compositionality of natural languages, allowing them to be used independently as "text tokens" or combined with natural language tokens to prompt pretrained T2I models with both visual and textual inputs, mirroring how we interact with vision-language models (VLMs). Experiments demonstrate that ViLex achieves higher fidelity in image reconstruction compared to text embeddings--even with a single ViLex token. Moreover, ViLex successfully performs various DreamBooth tasks in a zero-shot, unsupervised manner without fine-tuning T2I models. Additionally, ViLex serves as a powerful vision encoder, consistently improving vision-language model performance across 15 benchmarks relative to a strong SigLIP baseline.