Personality-aware Human-centric Multimodal Reasoning: A New Task, Dataset and Baselines

TL;DR

This paper introduces a new task called Personality-aware Human-centric Multimodal Reasoning (PHMR) that predicts future individual behavior by integrating personality traits with multimodal data, supported by a new dataset and baseline methods.

Contribution

It proposes the first task to incorporate personality traits into multimodal reasoning for behavior prediction, along with a new dataset, baseline models, and an extension for personality prediction.

Findings

Incorporating personality traits improves reasoning performance.

The new dataset contains 12,000 samples from TV shows.

Baseline models demonstrate the effectiveness of personality integration.

Abstract

Personality traits, emotions, and beliefs shape individuals' behavioral choices and decision-making processes. However, for one thing, the affective computing community normally focused on predicting personality traits but overlooks their application in behavior prediction. For another, the multimodal reasoning task emphasized the prediction of future states and behaviors but often neglected the incorporation of individual personality traits. In this work, we introduce a new task called Personality-aware Human-centric Multimodal Reasoning (PHMR) (T1), with the goal of forecasting the future behavior of a particular individual using multimodal information from past instances, while integrating personality factors. We accordingly construct a new dataset based on six television shows, encompassing 225 characters and 12k samples. To establish a benchmark for the task, we propose seven baseline methods: three adapted from related tasks, two pre-trained model, and two multimodal large language models. The experimental results demonstrate that incorporating personality traits enhances human-centric multimodal reasoning performance. To further solve the lack of personality annotation in real-life scenes, we introduce an extension task called Personality-predicted Human-centric Multimodal Reasoning task (T2) along with the corresponding dataset and method. We will make our dataset and code available on GitHub.