LexSemBridge: Fine-Grained Dense Representation Enhancement through Token-Aware Embedding Augmentation

TL;DR

LexSemBridge enhances dense retrieval models by incorporating fine-grained, input-aware vector modulation to improve performance on challenging, fine-grained retrieval tasks in both text and vision domains.

Contribution

Introduces LexSemBridge, a novel framework that improves dense representations with input-aware vector modulation using three paradigms, without altering backbone encoders.

Findings

Significant improvements on fine-grained retrieval tasks

Effective across both text and vision modalities

Plug-in design preserves semantic integrity

Abstract

As queries in retrieval-augmented generation (RAG) pipelines powered by large language models (LLMs) become increasingly complex and diverse, dense retrieval models have demonstrated strong performance in semantic matching. Nevertheless, they often struggle with fine-grained retrieval tasks, where precise keyword alignment and span-level localization are required, even in cases with high lexical overlap that would intuitively suggest easier retrieval. To systematically evaluate this limitation, we introduce two targeted tasks, keyword retrieval and part-of-passage retrieval, designed to simulate practical fine-grained scenarios. Motivated by these observations, we propose LexSemBridge, a unified framework that enhances dense query representations through fine-grained, input-aware vector modulation. LexSemBridge constructs latent enhancement vectors from input tokens using three paradigms: Statistical (SLR), Learned (LLR), and Contextual (CLR), and integrates them with dense embeddings via element-wise interaction. Theoretically, we show that this modulation preserves the semantic direction while selectively amplifying discriminative dimensions. LexSemBridge operates as a plug-in without modifying the backbone encoder and naturally extends to both text and vision modalities. Extensive experiments across semantic and fine-grained retrieval tasks validate the effectiveness and generality of our approach. All code and models are publicly available at https://github.com/Jasaxion/LexSemBridge/