Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization

TL;DR

This paper introduces a novel approach for multimodal abstractive summarization that emphasizes summary-oriented visual features, utilizing auxiliary tasks to enhance performance across diverse resource scenarios and establishing a new multilingual dataset.

Contribution

It proposes a new training framework with auxiliary tasks to capture summary-oriented visual features, improving MAS performance especially in low-resource settings.

Findings

Achieves state-of-the-art results across 44 languages.

Effective in low- and zero-resource scenarios.

Provides a large-scale multilingual multimodal dataset.

Abstract

Multimodal abstractive summarization (MAS) aims to produce a concise summary given the multimodal data (text and vision). Existing studies mainly focus on how to effectively use the visual features from the perspective of an article, having achieved impressive success on the high-resource English dataset. However, less attention has been paid to the visual features from the perspective of the summary, which may limit the model performance, especially in the low- and zero-resource scenarios. In this paper, we propose to improve the summary quality through summary-oriented visual features. To this end, we devise two auxiliary tasks including vision to summary task and masked image modeling task. Together with the main summarization task, we optimize the MAS model via the training objectives of all these tasks. By these means, the MAS model can be enhanced by capturing the summary-oriented visual features, thereby yielding more accurate summaries. Experiments on 44 languages, covering mid-high-, low-, and zero-resource scenarios, verify the effectiveness and superiority of the proposed approach, which achieves state-of-the-art performance under all scenarios. Additionally, we will contribute a large-scale multilingual multimodal abstractive summarization (MM-Sum) dataset.