# Insights into Object Semantics: Leveraging Transformer Networks for Advanced Image Captioning

## TL;DR

This paper introduces a new image captioning model that uses Transformer networks and semantic object information to generate more accurate and diverse image descriptions.

## Contribution

The novel contribution is integrating instance-level semantic concepts into the encoder's attention mechanism to enhance visual feature representation.

## Key findings

- The model achieves state-of-the-art performance on the MS-COCO dataset.
- Incorporating semantic object information improves caption accuracy and diversity.
- The Transformer-based decoder outperforms traditional RNN-based approaches in language generation.

## Abstract

Image captioning is a technique used to generate descriptive captions for images. Typically, it involves employing a Convolutional Neural Network (CNN) as the encoder to extract visual features, and a decoder model, often based on Recurrent Neural Networks (RNNs), to generate the captions. Recently, the encoder–decoder architecture has witnessed the widespread adoption of the self-attention mechanism. However, this approach faces certain challenges that require further research. One such challenge is that the extracted visual features do not fully exploit the available image information, primarily due to the absence of semantic concepts. This limitation restricts the ability to fully comprehend the content depicted in the image. To address this issue, we present a new image-Transformer-based model boosted with image object semantic representation. Our model incorporates semantic representation in encoder attention, enhancing visual features by integrating instance-level concepts. Additionally, we employ Transformer as the decoder in the language generation module. By doing so, we achieve improved performance in generating accurate and diverse captions. We evaluated the performance of our model on the MS-COCO and novel MACE datasets. The results illustrate that our model aligns with state-of-the-art approaches in terms of caption generation.