IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning

TL;DR

IFCap introduces a unified zero-shot captioning framework that aligns text and image features and filters entities to improve caption quality without paired training data.

Contribution

The paper proposes Image-like Retrieval and Frequency-based Entity Filtering techniques integrated into a new framework for zero-shot captioning, addressing modality gaps and enhancing accuracy.

Findings

Outperforms state-of-the-art methods in image captioning

Achieves significant improvements in video captioning

Effective in zero-shot settings without paired data

Abstract

Recent advancements in image captioning have explored text-only training methods to overcome the limitations of paired image-text data. However, existing text-only training methods often overlook the modality gap between using text data during training and employing images during inference. To address this issue, we propose a novel approach called Image-like Retrieval, which aligns text features with visually relevant features to mitigate the modality gap. Our method further enhances the accuracy of generated captions by designing a Fusion Module that integrates retrieved captions with input features. Additionally, we introduce a Frequency-based Entity Filtering technique that significantly improves caption quality. We integrate these methods into a unified framework, which we refer to as IFCap ($\textbf{I}$mage-like Retrieval and $\textbf{F}$requency-based Entity Filtering for Zero-shot $\textbf{Cap}$tioning). Through extensive experimentation, our straightforward yet powerful approach has demonstrated its efficacy, outperforming the state-of-the-art methods by a significant margin in both image captioning and video captioning compared to zero-shot captioning based on text-only training.