Retrieval-based Disentangled Representation Learning with Natural Language Supervision

TL;DR

This paper introduces VDR, a retrieval-based framework that uses natural language descriptions to learn disentangled data representations, improving retrieval performance and interpretability across multiple datasets.

Contribution

VDR leverages natural language as a proxy for data variation, enabling disentangled representation learning through a bi-encoder model in a vocabulary space, with extensive benchmark evaluation.

Findings

Achieves 8.7% improvement in NDCG@10 on BEIR

Outperforms previous methods in mean recall on MS COCO and Flickr30k

Human evaluation shows interpretability comparable to SOTA captioning models

Abstract

Disentangled representation learning remains challenging as the underlying factors of variation in the data do not naturally exist. The inherent complexity of real-world data makes it unfeasible to exhaustively enumerate and encapsulate all its variations within a finite set of factors. However, it is worth noting that most real-world data have linguistic equivalents, typically in the form of textual descriptions. These linguistic counterparts can represent the data and effortlessly decomposed into distinct tokens. In light of this, we present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning. Our approach employ a bi-encoder model to represent both data and natural language in a vocabulary space, enabling the model to distinguish dimensions that capture intrinsic characteristics within data through its natural language counterpart, thus facilitating disentanglement. We extensively assess the performance of VDR across 15 retrieval benchmark datasets, covering text-to-text and cross-modal retrieval scenarios, as well as human evaluation. Our experimental results compellingly demonstrate the superiority of VDR over previous bi-encoder retrievers with comparable model size and training costs, achieving an impressive 8.7% improvement in NDCG@10 on the BEIR benchmark, a 5.3% increase on MS COCO, and a 6.0% increase on Flickr30k in terms of mean recall in the zero-shot setting. Moreover, The results from human evaluation indicate that interpretability of our method is on par with SOTA captioning models.