User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning

TL;DR

This paper introduces a user-aware prefix-tuning framework for personalized image captioning that efficiently leverages user context and large language models, achieving significant improvements over existing methods.

Contribution

The proposed framework uniquely combines prefix-tuning with frozen large language models to enhance personalized image captioning without extensive retraining.

Findings

Outperforms baseline models on Instagram and YFCC100M datasets

Achieves twofold improvements in BLEU-4 and CIDEr metrics

Reduces training parameters and computational costs

Abstract

Image captioning bridges the gap between vision and language by automatically generating natural language descriptions for images. Traditional image captioning methods often overlook the preferences and characteristics of users. Personalized image captioning solves this problem by incorporating user prior knowledge into the model, such as writing styles and preferred vocabularies. Most existing methods emphasize the user context fusion process by memory networks or transformers. However, these methods ignore the distinct domains of each dataset. Therefore, they need to update the entire caption model parameters when meeting new samples, which is time-consuming and calculation-intensive. To address this challenge, we propose a novel personalized image captioning framework that leverages user context to consider personality factors. Additionally, our framework utilizes the prefix-tuning paradigm to extract knowledge from a frozen large language model, reducing the gap between different language domains. Specifically, we employ CLIP to extract the visual features of an image and align the semantic space using a query-guided mapping network. By incorporating the transformer layer, we merge the visual features with the user's contextual prior knowledge to generate informative prefixes. Moreover, we employ GPT-2 as the frozen large language model. With a small number of parameters to be trained, our model performs efficiently and effectively. Our model outperforms existing baseline models on Instagram and YFCC100M datasets across five evaluation metrics, demonstrating its superiority, including twofold improvements in metrics such as BLEU-4 and CIDEr.