A Chain-of-Thought Subspace Meta-Learning for Few-shot Image Captioning with Large Vision and Language Models

TL;DR

This paper introduces a chain-of-thought subspace meta-learning approach for few-shot image captioning that improves the ability of large vision and language models to generate accurate descriptions with limited data.

Contribution

It proposes a multi-step chain-of-thought meta-learning scheme with subspace parameter learning to enhance few-shot image captioning performance.

Findings

Outperforms baseline methods on MSCOCO, Flickr8k, and Flickr30k datasets.

Demonstrates improved accuracy in few-shot image captioning tasks.

Validates the effectiveness of chain-of-thought meta-learning in multimodal settings.

Abstract

A large-scale vision and language model that has been pretrained on massive data encodes visual and linguistic prior, which makes it easier to generate images and language that are more natural and realistic. Despite this, there is still a significant domain gap between the modalities of vision and language, especially when training data is scarce in few-shot settings, where only very limited data are available for training. In order to mitigate this issue, a multi-modal meta-learning framework has been proposed to bridge the gap between two frozen pretrained large vision and language models by introducing a tunable prompt connecting these two large models. For few-shot image captioning, the existing multi-model meta-learning framework utilizes a one-step prompting scheme to accumulate the visual features of input images to guide the language model, which struggles to generate accurate image descriptions with only a few training samples. Instead, we propose a chain-of-thought (CoT) meta-learning scheme as a multi-step image captioning procedure to better imitate how humans describe images. In addition, we further propose to learn different meta-parameters of the model corresponding to each CoT step in distinct subspaces to avoid interference. We evaluated our method on three commonly used image captioning datasets, i.e., MSCOCO, Flickr8k, and Flickr30k, under few-shot settings. The results of our experiments indicate that our chain-of-thought subspace meta-learning strategy is superior to the baselines in terms of performance across different datasets measured by different metrics.