Affective Feedback Synthesis Towards Multimodal Text and Image Data

TL;DR

This paper introduces a new task of affective feedback synthesis for multimodal data, proposing a system trained on a large dataset of images, texts, and human comments, to generate human-like feedback.

Contribution

It presents a novel multimodal feedback synthesis system and constructs a large-scale dataset for training and evaluation.

Findings

Generated feedbacks are semantically similar to ground-truth comments.

The system produces relevant and human-like responses.

Quantitative and qualitative evaluations show improved performance.

Abstract

In this paper, we have defined a novel task of affective feedback synthesis that deals with generating feedback for input text & corresponding image in a similar way as humans respond towards the multimodal data. A feedback synthesis system has been proposed and trained using ground-truth human comments along with image-text input. We have also constructed a large-scale dataset consisting of image, text, Twitter user comments, and the number of likes for the comments by crawling the news articles through Twitter feeds. The proposed system extracts textual features using a transformer-based textual encoder while the visual features have been extracted using a Faster region-based convolutional neural networks model. The textual and visual features have been concatenated to construct the multimodal features using which the decoder synthesizes the feedback. We have compared the results of the proposed system with the baseline models using quantitative and qualitative measures. The generated feedbacks have been analyzed using automatic and human evaluation. They have been found to be semantically similar to the ground-truth comments and relevant to the given text-image input.