Learning complete and explainable visual representations from itemized text supervision

TL;DR

This paper introduces ItemizedCLIP, a framework that learns complete, explainable, and semantically grounded visual representations from itemized text supervision across various domains, improving zero-shot and interpretability performance.

Contribution

The paper proposes ItemizedCLIP, a novel method that effectively leverages itemized text annotations to produce comprehensive and interpretable visual embeddings, addressing a gap in non-object-centric domain supervision.

Findings

Significant zero-shot performance improvements across multiple domains.

Enhanced interpretability and item-differentiability of visual representations.

Effective modeling of item independence and coverage in visual embeddings.

Abstract

Training vision models with language supervision enables general and transferable representations. However, many visual domains, especially non-object-centric domains such as medical imaging and remote sensing, contain itemized text annotations: multiple text items describing distinct and semantically independent findings within a single image. Such supervision differs from standard multi-caption supervision, where captions are redundant or highly overlapping. Here, we introduce ItemizedCLIP, a framework for learning complete and explainable visual representations from itemized text supervision. ItemizedCLIP employs a cross-attention module to produce text item-conditioned visual embeddings and a set of tailored objectives that jointly enforce item independence (distinct regions for distinct items) and representation completeness (coverage of all items). Across four domains with naturally itemized text supervision (brain MRI, head CT, chest CT, remote sensing) and one additional synthetically itemized dataset, ItemizedCLIP achieves substantial improvements in zero-shot performance and fine-grained interpretability over baselines. The resulting ItemizedCLIP representations are semantically grounded, item-differentiable, complete, and visually interpretable. Our code is available at https://github.com/MLNeurosurg/ItemizedCLIP.