Spatial Attention as an Interface for Image Captioning Models

TL;DR

This paper explores translating spatial attention mechanisms in image captioning models into natural language to better understand their internal visual functions, demonstrating that attention manipulations influence generated captions and can reveal model focus.

Contribution

It introduces methods to interpret spatial attention in image captioning models as natural language interfaces, linking attention to model behavior and extending to visual question answering networks.

Findings

Model reacts to attention modifications with up to 52.65% change in caption content.

Generated captions include object details in 9% of cases when attention is manipulated.

Captions reflect question-answer details in 55.20% of cases at the word level.

Abstract

The internal workings of modern deep learning models stay often unclear to an external observer, although spatial attention mechanisms are involved. The idea of this work is to translate these spatial attentions into natural language to provide a simpler access to the model's function. Thus, I took a neural image captioning model and measured the reactions to external modification in its spatial attention for three different interface methods: a fixation over the whole generation process, a fixation for the first time-steps and an addition to the generator's attention. The experimental results for bounding box based spatial attention vectors have shown that the captioning model reacts to method dependent changes in up to 52.65% and includes in 9.00% of the cases object categories, which were otherwise unmentioned. Afterwards, I established such a link to a hierarchical co-attention network for visual question answering by extraction of its word, phrase and question level spatial attentions. Here, generated captions for the word level included details of the question-answer pairs in up to 55.20% of the cases. This work indicates that spatial attention seen as an external interface for image caption generators is an useful method to access visual functions in natural language.