An analysis of vision-language models for fabric retrieval

TL;DR

This paper explores how vision-language models can improve fabric retrieval by using structured descriptions generated via multimodal large language models, highlighting the importance of domain-specific adaptations.

Contribution

Introduces an automated annotation pipeline using MLLMs to generate structured descriptions for fabric retrieval, and evaluates multiple VLMs in a specialized domain.

Findings

Structured descriptions improve retrieval accuracy.

Perception Encoder outperforms other models.

Zero-shot retrieval remains challenging in fine-grained domains.

Abstract

Effective cross-modal retrieval is essential for applications like information retrieval and recommendation systems, particularly in specialized domains such as manufacturing, where product information often consists of visual samples paired with a textual description. This paper investigates the use of Vision Language Models(VLMs) for zero-shot text-to-image retrieval on fabric samples. We address the lack of publicly available datasets by introducing an automated annotation pipeline that uses Multimodal Large Language Models (MLLMs) to generate two types of textual descriptions: freeform natural language and structured attribute-based descriptions. We produce these descriptions to evaluate retrieval performance across three Vision-Language Models: CLIP, LAION-CLIP, and Meta's Perception Encoder. Our experiments demonstrate that structured, attribute-rich descriptions significantly enhance retrieval accuracy, particularly for visually complex fabric classes, with the Perception Encoder outperforming other models due to its robust feature alignment capabilities. However, zero-shot retrieval remains challenging in this fine-grained domain, underscoring the need for domain-adapted approaches. Our findings highlight the importance of combining technical textual descriptions with advanced VLMs to optimize cross-modal retrieval in industrial applications.