Large-Scale Bidirectional Training for Zero-Shot Image Captioning

TL;DR

This paper introduces BITTERS, a large-scale bidirectional training framework that enables zero-shot image captioning, along with a new benchmark for evaluation and an efficient finetuning method for keyword extraction.

Contribution

The paper presents a novel large-scale bidirectional training approach for zero-shot image captioning and a comprehensive evaluation benchmark.

Findings

Bidirectional training improves zero-shot captioning accuracy.

Careful dataset and architecture selection are crucial.

Proposed finetuning method enhances keyword extraction.

Abstract

When trained on large-scale datasets, image captioning models can understand the content of images from a general domain but often fail to generate accurate, detailed captions. To improve performance, pretraining-and-finetuning has been a key strategy for image captioning. However, we find that large-scale bidirectional training between image and text enables zero-shot image captioning. In this paper, we introduce Bidirectional Image Text Training in largER Scale, BITTERS, an efficient training and inference framework for zero-shot image captioning. We also propose a new evaluation benchmark which comprises of high quality datasets and an extensive set of metrics to properly evaluate zero-shot captioning accuracy and societal bias. We additionally provide an efficient finetuning approach for keyword extraction. We show that careful selection of large-scale training set and model architecture is the key to achieving zero-shot image captioning.