On the Potential of CLIP for Compositional Logical Reasoning

TL;DR

This paper investigates whether CLIP's embeddings can support logically coherent visual reasoning and finds that, in its standard form, it cannot perform such reasoning effectively.

Contribution

The paper provides a formal geometric analysis of CLIP's embedding space and demonstrates its limitations for compositional logical reasoning.

Findings

CLIP's embeddings are not configured for logical coherence

Standard CLIP cannot perform grounded logical reasoning

Geometric analysis explains CLIP's reasoning limitations

Abstract

In this paper we explore the possibility of using OpenAI's CLIP to perform logically coherent grounded visual reasoning. To that end, we formalize our terms and give a geometric analysis of how embeddings in CLIP's latent space would need to be configured in order for the system to be logically coherent. Our main conclusion is that, as usually configured, CLIP cannot perform such reasoning.