With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning

TL;DR

This paper introduces a prototypical memory network for image captioning that enhances Transformer models by incorporating semantic information from other samples, leading to improved performance on the COCO dataset.

Contribution

The paper proposes a novel prototypical memory mechanism for attention in image captioning, capturing semantic information from multiple samples to boost Transformer performance.

Findings

Achieved a 3.7 CIDEr points improvement on COCO dataset

Enhanced Transformer-based captioning with sample-aware attention

Demonstrated effectiveness of prototype-based memory in vision-language tasks

Abstract

Image captioning, like many tasks involving vision and language, currently relies on Transformer-based architectures for extracting the semantics in an image and translating it into linguistically coherent descriptions. Although successful, the attention operator only considers a weighted summation of projections of the current input sample, therefore ignoring the relevant semantic information which can come from the joint observation of other samples. In this paper, we devise a network which can perform attention over activations obtained while processing other training samples, through a prototypical memory model. Our memory models the distribution of past keys and values through the definition of prototype vectors which are both discriminative and compact. Experimentally, we assess the performance of the proposed model on the COCO dataset, in comparison with carefully designed baselines and state-of-the-art approaches, and by investigating the role of each of the proposed components. We demonstrate that our proposal can increase the performance of an encoder-decoder Transformer by 3.7 CIDEr points both when training in cross-entropy only and when fine-tuning with self-critical sequence training. Source code and trained models are available at: https://github.com/aimagelab/PMA-Net.