PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis

TL;DR

This paper introduces PanoSent, a comprehensive benchmark and framework for multimodal conversational aspect-based sentiment analysis, focusing on recognizing detailed sentiment elements and their dynamic changes in multi-party dialogues.

Contribution

It proposes the novel tasks of Panoptic Sentiment Sextuple Extraction and Sentiment Flipping Analysis, along with a new dataset, a chain-of-sentiment reasoning framework, and a multimodal large language model, Sentica.

Findings

Our methods outperform strong baselines in all tasks.

The dataset covers diverse languages, scenarios, and sentiment types.

The framework effectively captures sentiment dynamics and causal rationales.

Abstract

While existing Aspect-based Sentiment Analysis (ABSA) has received extensive effort and advancement, there are still gaps in defining a more holistic research target seamlessly integrating multimodality, conversation context, fine-granularity, and also covering the changing sentiment dynamics as well as cognitive causal rationales. This paper bridges the gaps by introducing a multimodal conversational ABSA, where two novel subtasks are proposed: 1) Panoptic Sentiment Sextuple Extraction, panoramically recognizing holder, target, aspect, opinion, sentiment, rationale from multi-turn multi-party multimodal dialogue. 2) Sentiment Flipping Analysis, detecting the dynamic sentiment transformation throughout the conversation with the causal reasons. To benchmark the tasks, we construct PanoSent, a dataset annotated both manually and automatically, featuring high quality, large scale, multimodality, multilingualism, multi-scenarios, and covering both implicit and explicit sentiment elements. To effectively address the tasks, we devise a novel Chain-of-Sentiment reasoning framework, together with a novel multimodal large language model (namely Sentica) and a paraphrase-based verification mechanism. Extensive evaluations demonstrate the superiority of our methods over strong baselines, validating the efficacy of all our proposed methods. The work is expected to open up a new era for the ABSA community, and thus all our codes and data are open at https://PanoSent.github.io/