The Visual Language of Fabrics

TL;DR

This paper introduces text2fabric, a dataset linking free-text descriptions to fabric images, enabling better understanding and retrieval of fabric materials through natural language, and improving vision-language models' performance in this domain.

Contribution

The creation of the text2fabric dataset linking natural language descriptions to fabric images and analysis of fabric description structure for improved material understanding.

Findings

Dataset enables fine-grained fabric retrieval.

Improves automatic fabric captioning.

Enhances vision-language model specialization.

Abstract

We introduce text2fabric, a novel dataset that links free-text descriptions to various fabric materials. The dataset comprises 15,000 natural language descriptions associated to 3,000 corresponding images of fabric materials. Traditionally, material descriptions come in the form of tags/keywords, which limits their expressivity, induces pre-existing knowledge of the appropriate vocabulary, and ultimately leads to a chopped description system. Therefore, we study the use of free-text as a more appropriate way to describe material appearance, taking the use case of fabrics as a common item that non-experts may often deal with. Based on the analysis of the dataset, we identify a compact lexicon, set of attributes and key structure that emerge from the descriptions. This allows us to accurately understand how people describe fabrics and draw directions for generalization to other types of materials. We also show that our dataset enables specializing large vision-language models such as CLIP, creating a meaningful latent space for fabric appearance, and significantly improving applications such as fine-grained material retrieval and automatic captioning.