Synthesizing Sentiment-Controlled Feedback For Multimodal Text and Image Data

TL;DR

This paper introduces a large-scale dataset and a system for generating sentiment-controlled feedback for multimodal data, improving the ability of AI to produce empathetic responses involving text and images.

Contribution

It presents the CMFeed dataset and a novel controllable feedback synthesis system that effectively generates sentiment-specific responses for multimodal inputs.

Findings

Sentiment classification accuracy reached 77.23%.

The system outperforms baseline models by 18.82%.

The dataset includes multimodal inputs and human-annotated reactions.

Abstract

The ability to generate sentiment-controlled feedback in response to multimodal inputs comprising text and images addresses a critical gap in human-computer interaction. This capability allows systems to provide empathetic, accurate, and engaging responses, with useful applications in education, healthcare, marketing, and customer service. To this end, we have constructed a large-scale Controllable Multimodal Feedback Synthesis (CMFeed) dataset and proposed a controllable feedback synthesis system. The system features an encoder, decoder, and controllability block for textual and visual inputs. It extracts features using a transformer and a Faster R-CNN network, combining them to generate feedback. The CMFeed dataset includes images, texts, reactions to the posts, human comments with relevance scores, and reactions to these comments. These reactions train the model to produce feedback with specified sentiments, achieving a sentiment classification accuracy of 77.23%, which is 18.82% higher than the accuracy without controllability. Access to the CMFeed dataset and the system's code is available at https://github.com/MIntelligence-Group/CMFeed.