ZoDIAC: Zoneout Dropout Injection Attention Calculation

TL;DR

ZoDIAC introduces a novel attention refinement and intensification mechanism for transformers, significantly improving image captioning performance by injecting learned scalar factors into self-attention calculations.

Contribution

The paper proposes ZoDIAC, a new attention mechanism that refines and intensifies self-attention in transformers using GELU, dropout, and a learned scalar, enhancing performance in image captioning.

Findings

ZoDIAC outperforms conventional self-attention on MS-COCO.

Statistically significant improvements across all captioning metrics.

ZoDIAC can replace standard attention modules in various transformer models.

Abstract

In the past few years the transformer model has been utilized for a variety of tasks such as image captioning, image classification natural language generation, and natural language understanding. As a key component of the transformer model, self-attention calculates the attention values by mapping the relationships among the head elements of the source and target sequence, yet there is no explicit mechanism to refine and intensify the attention values with respect to the context of the input and target sequences. Based on this intuition, we introduce a novel refine and intensify attention mechanism that is called Zoneup Dropout Injection Attention Calculation (ZoDIAC), in which the intensities of attention values in the elements of the input source and target sequences are first refined using GELU and dropout and then intensified using a proposed zoneup process which includes the injection of a learned scalar factor. Our extensive experiments show that ZoDIAC achieves statistically significant higher scores under all image captioning metrics using various feature extractors in comparison to the conventional self-attention module in the transformer model on the MS-COCO dataset. Our proposed ZoDIAC attention modules can be used as a drop-in replacement for the attention components in all transformer models. The code for our experiments is publicly available at: https://github.com/zanyarz/zodiac