Language-Agnostic Visual Embeddings for Cross-Script Handwriting Retrieval

TL;DR

This paper introduces a lightweight, language-agnostic visual embedding framework for cross-script handwriting retrieval, achieving state-of-the-art accuracy with fewer parameters and enabling efficient cross-lingual retrieval.

Contribution

The authors propose a novel asymmetric dual-encoder model that learns style-invariant, language-agnostic visual embeddings for handwriting retrieval, addressing computational and cross-lingual challenges.

Findings

Outperforms 28 baseline methods in within-language retrieval

Achieves state-of-the-art accuracy on handwriting retrieval benchmarks

Enables effective cross-lingual handwriting retrieval with fewer parameters

Abstract

Handwritten word retrieval is vital for digital archives but remains challenging due to large handwriting variability and cross-lingual semantic gaps. While large vision-language models offer potential solutions, their prohibitive computational costs hinder practical edge deployment. To address this, we propose a lightweight asymmetric dual-encoder framework that learns unified, style-invariant visual embeddings. By jointly optimizing instance-level alignment and class-level semantic consistency, our approach anchors visual embeddings to language-agnostic semantic prototypes, enforcing invariance across scripts and writing styles. Experiments show that our method outperforms 28 baselines and achieves state-of-the-art accuracy on within-language retrieval benchmarks. We further conduct explicit cross-lingual retrieval, where the query language differs from the target language, to validate the effectiveness of the learned cross-lingual representations. Achieving strong performance with only a fraction of the parameters required by existing models, our framework enables accurate and resource-efficient cross-script handwriting retrieval.