Enhanced Modality Transition for Image Captioning

TL;DR

This paper introduces a Modality Transition Module (MTM) that transforms visual features into semantic representations to improve image captioning, resulting in more detailed and contextually accurate captions.

Contribution

The paper proposes a novel MTM that explicitly bridges the modality gap in image captioning, enhancing caption quality over existing encoder-decoder models.

Findings

Improves captioning performance by 3.4% on MS-COCO.

Effectively transfers visual features into semantic space.

Enhances contextual and detailed caption generation.

Abstract

Image captioning model is a cross-modality knowledge discovery task, which targets at automatically describing an image with an informative and coherent sentence. To generate the captions, the previous encoder-decoder frameworks directly forward the visual vectors to the recurrent language model, forcing the recurrent units to generate a sentence based on the visual features. Although these sentences are generally readable, they still suffer from the lack of details and highlights, due to the fact that the substantial gap between the image and text modalities is not sufficiently addressed. In this work, we explicitly build a Modality Transition Module (MTM) to transfer visual features into semantic representations before forwarding them to the language model. During the training phase, the modality transition network is optimised by the proposed modality loss, which compares the generated preliminary textual encodings with the target sentence vectors from a pre-trained text auto-encoder. In this way, the visual vectors are transited into the textual subspace for more contextual and precise language generation. The novel MTM can be incorporated into most of the existing methods. Extensive experiments have been conducted on the MS-COCO dataset demonstrating the effectiveness of the proposed framework, improving the performance by 3.4% comparing to the state-of-the-arts.