WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

TL;DR

WikiCLIP introduces an efficient contrastive approach for open-domain visual entity recognition, leveraging language and visual cues to outperform existing methods with significantly lower latency.

Contribution

It presents WikiCLIP, a novel framework combining language embeddings, visual alignment, and negative synthesis for improved accuracy and efficiency in VER.

Findings

Achieves 16% improvement on OVEN unseen set.

Reduces inference latency by nearly 100 times.

Outperforms strong baselines on open-domain VER benchmarks.

Abstract

Open-domain visual entity recognition (VER) seeks to associate images with entities in encyclopedic knowledge bases such as Wikipedia. Recent generative methods tailored for VER demonstrate strong performance but incur high computational costs, limiting their scalability and practical deployment. In this work, we revisit the contrastive paradigm for VER and introduce WikiCLIP, a simple yet effective framework that establishes a strong and efficient baseline for open-domain VER. WikiCLIP leverages large language model embeddings as knowledge-rich entity representations and enhances them with a Vision-Guided Knowledge Adaptor (VGKA) that aligns textual semantics with visual cues at the patch level. To further encourage fine-grained discrimination, a Hard Negative Synthesis Mechanism generates visually similar but semantically distinct negatives during training. Experimental results on popular open-domain VER benchmarks, such as OVEN, demonstrate that WikiCLIP significantly outperforms strong baselines. Specifically, WikiCLIP achieves a 16\% improvement on the challenging OVEN unseen set, while reducing inference latency by nearly 100 times compared with the leading generative model, AutoVER. The project page is available at https://artanic30.github.io/project_pages/WikiCLIP/