CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization

TL;DR

CISum introduces a cross-modality learning framework that enhances multimodal summarization by translating images into descriptions, fusing them with text, and selecting relevant images, leading to improved semantic coverage.

Contribution

The paper proposes a novel multi-task learning approach that explicitly models cross-modality interactions to improve semantic coverage in multimodal summarization.

Findings

CISum outperforms baselines in multimodal semantic coverage metrics.

CISum maintains high ROUGE and BLEU scores.

Introduces an automatic multimodal semantics coverage metric.

Abstract

Multimodal summarization (MS) aims to generate a summary from multimodal input. Previous works mainly focus on textual semantic coverage metrics such as ROUGE, which considers the visual content as supplemental data. Therefore, the summary is ineffective to cover the semantics of different modalities. This paper proposes a multi-task cross-modality learning framework (CISum) to improve multimodal semantic coverage by learning the cross-modality interaction in the multimodal article. To obtain the visual semantics, we translate images into visual descriptions based on the correlation with text content. Then, the visual description and text content are fused to generate the textual summary to capture the semantics of the multimodal content, and the most relevant image is selected as the visual summary. Furthermore, we design an automatic multimodal semantics coverage metric to evaluate the performance. Experimental results show that CISum outperforms baselines in multimodal semantics coverage metrics while maintaining the excellent performance of ROUGE and BLEU.