A New Method to Capturing Compositional Knowledge in Linguistic Space

TL;DR

This paper introduces ZS-CU, a zero-shot method for compositional understanding in visual language models, using textual inversion and logical regularization to outperform state-of-the-art models without hard negative data.

Contribution

The paper presents YUKINO, a novel zero-shot approach leveraging textual inversion and logical regularization to enhance compositional understanding without hard negative training.

Findings

YUKINO outperforms SOTA models by over 8% on SugarCREPE benchmark.

YUKINO achieves significant improvements in image retrieval tasks.

The method reduces training complexity via knowledge distillation.

Abstract

Compositional understanding allows visual language models to interpret complex relationships between objects, attributes, and relations in images and text. However, most existing methods often rely on hard negative examples and fine-tuning, which can overestimate improvements and are limited by the difficulty of obtaining hard negatives. In this work, we introduce Zero-Shot Compositional Understanding (ZS-CU), a novel task that enhances compositional understanding without requiring hard negative training data. We propose YUKINO (Yielded Compositional Understanding Knowledge via Textual Inversion with NO), which uses textual inversion to map unlabeled images to pseudo-tokens in a pre-trained CLIP model. We propose introducing "no" logical regularization to address the issue of token interaction in inversion. Additionally, we suggest using knowledge distillation to reduce the time complexity of textual inversion. Experimental results show that YUKINO outperforms the existing multi-modal SOTA models by over 8% on the SugarCREPE benchmark, and also achieves significant improvements in image retrieval tasks.