CapText: Large Language Model-based Caption Generation From Image Context and Description

TL;DR

This paper introduces CapText, a novel method that uses large language models to generate image captions solely from textual descriptions and context, bypassing direct image processing.

Contribution

The paper presents a new approach leveraging large language models for caption generation from text-only input, outperforming existing image-text alignment models after fine-tuning.

Findings

Outperforms state-of-the-art models like OSCAR-VinVL on CIDEr metric

Effective caption generation using only textual descriptions and context

Demonstrates the potential of large language models in image captioning

Abstract

While deep-learning models have been shown to perform well on image-to-text datasets, it is difficult to use them in practice for captioning images. This is because captions traditionally tend to be context-dependent and offer complementary information about an image, while models tend to produce descriptions that describe the visual features of the image. Prior research in caption generation has explored the use of models that generate captions when provided with the images alongside their respective descriptions or contexts. We propose and evaluate a new approach, which leverages existing large language models to generate captions from textual descriptions and context alone, without ever processing the image directly. We demonstrate that after fine-tuning, our approach outperforms current state-of-the-art image-text alignment models like OSCAR-VinVL on this task on the CIDEr metric.