Learning to Retrieve User History and Generate User Profiles for Personalized Persuasiveness Prediction

TL;DR

This paper introduces a context-aware user profiling framework that improves persuasiveness prediction by generating optimal queries and summarizing user history, demonstrating significant performance gains on Reddit data.

Contribution

The paper presents a novel trainable framework with query generation and profiling components for personalized persuasiveness prediction, outperforming existing methods.

Findings

F1 score increased from 33% to 47% on Llama-3.3-70B-Instruct.

Effective profiles are context-dependent and predictor-specific.

User profiling enhances persuasiveness prediction accuracy.

Abstract

Estimating the persuasiveness of messages is critical in various applications, from recommender systems to safety assessment of LLMs. While it is imperative to consider the target persuadee's characteristics, such as their values, experiences, and reasoning styles, there is currently no established systematic framework to optimize leveraging a persuadee's past activities (e.g., conversations) to the benefit of a persuasiveness prediction model. To address this problem, we propose a context-aware user profiling framework with two trainable components: a query generator that generates optimal queries to retrieve persuasion-relevant records from a user's history, and a profiler that summarizes these records into a profile to effectively inform the persuasiveness prediction model. Our evaluation on the ChangeMyView Reddit dataset shows consistent improvements over existing methods across multiple predictor models, raising F1 from 33% to 47% on Llama-3.3-70B-Instruct. Further analysis shows that effective user profiles are context-dependent and predictor-specific, rather than relying on static attributes or surface-level similarity. Together, these results highlight the importance of task-oriented, context-dependent user profiling for personalized persuasiveness prediction.