GAME: Learning Multimodal Interactions via Graph Structures for Personality Trait Estimation

TL;DR

This paper introduces GAME, a multimodal graph-based encoder that integrates visual, auditory, and textual cues for accurate personality trait estimation from short videos, leveraging advanced neural networks and attention mechanisms.

Contribution

The paper presents a novel Graph-Augmented Multimodal Encoder (GAME) that effectively fuses multi-source features for personality prediction, combining graph structures, attention, and temporal modeling.

Findings

GAME outperforms existing methods on multiple benchmarks.

The proposed fusion module enhances multimodal integration.

The model demonstrates robustness and generalizability across datasets.

Abstract

Apparent personality analysis from short videos poses significant chal-lenges due to the complex interplay of visual, auditory, and textual cues. In this paper, we propose GAME, a Graph-Augmented Multimodal Encoder designed to robustly model and fuse multi-source features for automatic personality prediction. For the visual stream, we construct a facial graph and introduce a dual-branch Geo Two-Stream Network, which combines Graph Convolutional Networks (GCNs) and Convolutional Neural Net-works (CNNs) with attention mechanisms to capture both structural and appearance-based facial cues. Complementing this, global context and iden-tity features are extracted using pretrained ResNet18 and VGGFace back-bones. To capture temporal dynamics, frame-level features are processed by a BiGRU enhanced with temporal attention modules. Meanwhile, audio representations are derived from the VGGish network, and linguistic se-mantics are captured via the XLM-Roberta transformer. To achieve effective multimodal integration, we propose a Channel Attention-based Fusion module, followed by a Multi-Layer Perceptron (MLP) regression head for predicting personality traits. Extensive experiments show that GAME con-sistently outperforms existing methods across multiple benchmarks, vali-dating its effectiveness and generalizability.