Empathy Level Prediction in Multi-Modal Scenario with Supervisory Documentation Assistance

TL;DR

This paper presents a novel multi-modal empathy prediction framework that integrates video, audio, and text, utilizing supervisory documentation during training to improve empathy detection accuracy.

Contribution

The paper introduces a multi-modal empathy prediction method with supervisory documentation-assisted training, leveraging privileged information to enhance feature extraction and prediction performance.

Findings

Outperforms existing empathy prediction methods on multi-modal datasets.

Effectively incorporates privileged supervisory documents during training.

Achieves higher accuracy in empathy detection across modalities.

Abstract

Prevalent empathy prediction techniques primarily concentrate on a singular modality, typically textual, thus neglecting multi-modal processing capabilities. They also overlook the utilization of certain privileged information, which may encompass additional empathetic content. In response, we introduce an advanced multi-modal empathy prediction method integrating video, audio, and text information. The method comprises the Multi-Modal Empathy Prediction and Supervisory Documentation Assisted Training. We use pre-trained networks in the empathy prediction network to extract features from various modalities, followed by a cross-modal fusion. This process yields a multi-modal feature representation, which is employed to predict empathy labels. To enhance the extraction of text features, we incorporate supervisory documents as privileged information during the assisted training phase. Specifically, we apply the Latent Dirichlet Allocation model to identify potential topic distributions to constrain text features. These supervisory documents, created by supervisors, focus on the counseling topics and the counselor's display of empathy. Notably, this privileged information is only available during training and is not accessible during the prediction phase. Experimental results on the multi-modal and dialogue empathy datasets demonstrate that our approach is superior to the existing methods.