Netizen-Style Commenting on Fashion Photos: Dataset and Diversity Measures

TL;DR

This paper introduces Netizen Style Commenting (NSC) to generate culturally rich, engaging comments on fashion photos, supported by a large dataset and new diversity measures, improving captioning quality and engagement.

Contribution

It presents a novel framework combining topic models and neural networks for diverse, netizen-style comments, along with a large-scale dataset and diversity measures for fashion photo commenting.

Findings

Enhanced comment diversity and engagement in fashion photo commenting.

Improved accuracy and diversity in image captioning tasks.

Effective use of large-scale dataset for netizen-style comment generation.

Abstract

Recently, deep neural network models have achieved promising results in image captioning task. Yet, "vanilla" sentences, only describing shallow appearances (e.g., types, colors), generated by current works are not satisfied netizen style resulting in lacking engagements, contexts, and user intentions. To tackle this problem, we propose Netizen Style Commenting (NSC), to automatically generate characteristic comments to a user-contributed fashion photo. We are devoted to modulating the comments in a vivid "netizen" style which reflects the culture in a designated social community and hopes to facilitate more engagement with users. In this work, we design a novel framework that consists of three major components: (1) We construct a large-scale clothing dataset named NetiLook, which contains 300K posts (photos) with 5M comments to discover netizen-style comments. (2) We propose three unique measures to estimate the diversity of comments. (3) We bring diversity by marrying topic models with neural networks to make up the insufficiency of conventional image captioning works. Experimenting over Flickr30k and our NetiLook datasets, we demonstrate our proposed approaches benefit fashion photo commenting and improve image captioning tasks both in accuracy and diversity.