mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval

TL;DR

The paper introduces mEOL, a training-free, instruction-guided multimodal embedding framework that aligns text, images, and SVG code in a shared space, enhancing vector graphic and image retrieval without training.

Contribution

It presents a novel training-free method using MLLMs and SVG structural cues for multimodal embedding, outperforming traditional trained models in retrieval tasks.

Findings

mEOL outperforms encoder-based baselines in SVG and image retrieval.

The method enables prompt-level control over embeddings without training.

A new text-to-SVG retrieval benchmark demonstrates effectiveness.

Abstract

Scalable Vector Graphics (SVGs) function both as visual images and as structured code that encode rich geometric and layout information, yet most methods rasterize them and discard this symbolic organization. At the same time, recent sentence embedding methods produce strong text representations but do not naturally extend to visual or structured modalities. We propose a training-free, instruction-guided multimodal embedding framework that uses a Multimodal Large Language Model (MLLM) to map text, raster images, and SVG code into an aligned embedding space. We control the direction of embeddings through modality-specific instructions and structural SVG cues, eliminating the need for learned projection heads or contrastive training. Our method has two key components: (1) Multimodal Explicit One-word Limitation (mEOL), which instructs the MLLM to summarize any multimodal input into a single token whose hidden state serves as a compact semantic embedding. (2) A semantic SVG rewriting module that assigns meaningful identifiers and simplifies nested SVG elements through visual reasoning over the rendered image, exposing geometric and relational cues hidden in raw code. Using a repurposed VGBench, we build the first text-to-SVG retrieval benchmark and show that our training-free embeddings outperform encoder-based and training-based multimodal baselines. These results highlight prompt-level control as an effective alternative to parameter-level training for structure-aware multimodal retrieval. Project page: https://scene-the-ella.github.io/meol/