PersonaTAB: Predicting Personality Traits using Textual, Acoustic, and Behavioral Cues in Fully-Duplex Speech Dialogs

TL;DR

This paper introduces a system that predicts personality traits in spoken dialogues by combining textual, acoustic, and behavioral cues, enhancing personality-aware conversational agents.

Contribution

It presents a novel pipeline that preprocesses raw speech data to generate annotated datasets and employs large language models for personality prediction, addressing the lack of annotated speech datasets.

Findings

System achieves better alignment with human judgments than existing methods.

Pipeline effectively extracts and annotates speech data for personality prediction.

Combines multiple modalities for improved personality trait detection.

Abstract

Despite significant progress in neural spoken dialog systems, personality-aware conversation agents -- capable of adapting behavior based on personalities -- remain underexplored due to the absence of personality annotations in speech datasets. We propose a pipeline that preprocesses raw audio recordings to create a dialogue dataset annotated with timestamps, response types, and emotion/sentiment labels. We employ an automatic speech recognition (ASR) system to extract transcripts and timestamps, then generate conversation-level annotations. Leveraging these annotations, we design a system that employs large language models to predict conversational personality. Human evaluators were engaged to identify conversational characteristics and assign personality labels. Our analysis demonstrates that the proposed system achieves stronger alignment with human judgments compared to existing approaches.