Text-Guided Visual Representation Learning for Robust Multimodal E-Commerce Recommendation

TL;DR

This paper introduces TGQ-Former, a text-guided visual representation learning framework that enhances robustness in multimodal e-commerce retrieval by effectively handling noisy product images.

Contribution

It proposes a novel hybrid-query connector and a reliability-aware dual-gated module for improved visual token extraction guided by structured metadata.

Findings

TGQ-Former outperforms baselines on large-scale datasets.

Achieves a 6.04% improvement in Hit Rate@100.

Enhances robustness against promotional overlays and background clutter.

Abstract

Multimodal item embeddings are crucial for e-commerce item-to-item (I2I) retrieval, yet real-world product images often contain promotional overlays and background clutter that inject spurious visual cues and degrade retrieval robustness. This issue is particularly pronounced in MLRM-style pipelines, where a frozen vision encoder is connected to an LLM through a lightweight connector that must selectively aggregate visual tokens. We propose Text-Guided Q-Former (TGQ-Former), a text-guided visual representation learning framework that leverages structured metadata as semantic guidance for visual token extraction while preserving complementary visual evidence. Concretely, TGQ-Former employs a hybrid-query connector to disentangle metadata-anchored and exploratory visual streams, and introduces a lightweight reliability-aware dual-gated vector modulation module to adaptively calibrate their contributions under noisy inputs. Experiments on large-scale, real-world e-commerce datasets with full-pool retrieval show that TGQ-Former consistently outperforms strong connector baselines and end-to-end MLLMs. On average, it improves Hit Rate@100 (H@100) by 6.04%, demonstrating the effectiveness of text-guided visual encoding for robust multimodal retrieval.