ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic

TL;DR

ZeroCap leverages contrastive learning models combined with large language models to generate descriptive captions for images in a zero-shot manner, enabling high-level visual reasoning like image arithmetic and analogies without additional training.

Contribution

The paper introduces a novel zero-shot image captioning approach that combines visual-semantic models with language models, enabling flexible image-to-text generation and visual arithmetic tasks.

Findings

Generated captions are less restrictive than supervised methods.

Demonstrated ability to perform image arithmetic and visual analogies.

Achieved high flexibility in zero-shot image-to-text tasks.

Abstract

Recent text-to-image matching models apply contrastive learning to large corpora of uncurated pairs of images and sentences. While such models can provide a powerful score for matching and subsequent zero-shot tasks, they are not capable of generating caption given an image. In this work, we repurpose such models to generate a descriptive text given an image at inference time, without any further training or tuning steps. This is done by combining the visual-semantic model with a large language model, benefiting from the knowledge in both web-scale models. The resulting captions are much less restrictive than those obtained by supervised captioning methods. Moreover, as a zero-shot learning method, it is extremely flexible and we demonstrate its ability to perform image arithmetic in which the inputs can be either images or text, and the output is a sentence. This enables novel high-level vision capabilities such as comparing two images or solving visual analogy tests. Our code is available at: https://github.com/YoadTew/zero-shot-image-to-text.