MMAPS: End-to-End Multi-Grained Multi-Modal Attribute-Aware Product Summarization

TL;DR

This paper introduces MMAPS, an end-to-end multi-grained multi-modal attribute-aware model for product summarization that effectively integrates product attributes from text and images to generate high-quality summaries, outperforming existing methods.

Contribution

The paper presents a novel end-to-end multi-grained multi-modal approach that jointly models product attributes and summaries, addressing limitations of prior methods in multi-modal product summarization.

Findings

MMAPS outperforms state-of-the-art methods on a large-scale Chinese e-commerce dataset.

The model effectively integrates multi-modal attributes into product summaries.

Extensive experiments validate the superiority of MMAPS across multiple metrics.

Abstract

Given the long textual product information and the product image, Multi-modal Product Summarization (MPS) aims to increase customers' desire to purchase by highlighting product characteristics with a short textual summary. Existing MPS methods can produce promising results. Nevertheless, they still 1) lack end-to-end product summarization, 2) lack multi-grained multi-modal modeling, and 3) lack multi-modal attribute modeling. To improve MPS, we propose an end-to-end multi-grained multi-modal attribute-aware product summarization method (MMAPS) for generating high-quality product summaries in e-commerce. MMAPS jointly models product attributes and generates product summaries. We design several multi-grained multi-modal tasks to better guide the multi-modal learning of MMAPS. Furthermore, we model product attributes based on both text and image modalities so that multi-modal product characteristics can be manifested in the generated summaries. Extensive experiments on a real large-scale Chinese e-commence dataset demonstrate that our model outperforms state-of-the-art product summarization methods w.r.t. several summarization metrics. Our code is publicly available at: https://github.com/KDEGroup/MMAPS.