Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks

TL;DR

This paper introduces Linguistically-aware Attention (LAT), an attention mechanism that incorporates linguistic understanding into vision-language models to bridge the semantic gap, improving performance across tasks like VQA, counting, and captioning.

Contribution

The paper proposes LAT, a novel attention mechanism that integrates object attributes and language models to enhance linguistic awareness in vision-language tasks.

Findings

Achieved state-of-the-art results on five Counting-VQA datasets.

Consistently improved baseline models for VQA and image captioning.

Demonstrated the generic effectiveness of LAT across multiple tasks.

Abstract

Attention models are widely used in Vision-language (V-L) tasks to perform the visual-textual correlation. Humans perform such a correlation with a strong linguistic understanding of the visual world. However, even the best performing attention model in V-L tasks lacks such a high-level linguistic understanding, thus creating a semantic gap between the modalities. In this paper, we propose an attention mechanism - Linguistically-aware Attention (LAT) - that leverages object attributes obtained from generic object detectors along with pre-trained language models to reduce this semantic gap. LAT represents visual and textual modalities in a common linguistically-rich space, thus providing linguistic awareness to the attention process. We apply and demonstrate the effectiveness of LAT in three V-L tasks: Counting-VQA, VQA, and Image captioning. In Counting-VQA, we propose a novel counting-specific VQA model to predict an intuitive count and achieve state-of-the-art results on five datasets. In VQA and Captioning, we show the generic nature and effectiveness of LAT by adapting it into various baselines and consistently improving their performance.