Examining the Role of Relationship Alignment in Large Language Models

TL;DR

This study evaluates how well Llama 3.0 can predict and replicate social relationship-based semantic tones in comments, revealing strengths in semantic comprehension but limitations in personalization through prompting.

Contribution

It demonstrates that Llama 3.0 can understand semantics from posts and generate comments similar to humans, but struggles with personalized responses based on social context.

Findings

Including social info improves semantic tone prediction.

LLMs can understand semantics without social context in prompts.

Personalization via prompting remains limited.

Abstract

The rapid development and deployment of Generative AI in social settings raise important questions about how to optimally personalize them for users while maintaining accuracy and realism. Based on a Facebook public post-comment dataset, this study evaluates the ability of Llama 3.0 (70B) to predict the semantic tones across different combinations of a commenter's and poster's gender, age, and friendship closeness and to replicate these differences in LLM-generated comments. The study consists of two parts: Part I assesses differences in semantic tones across social relationship categories, and Part II examines the similarity between comments generated by Llama 3.0 (70B) and human comments from Part I given public Facebook posts as input. Part I results show that including social relationship information improves the ability of a model to predict the semantic tone of human comments. However, Part II results show that even without including social context information in the prompt, LLM-generated comments and human comments are equally sensitive to social context, suggesting that LLMs can comprehend semantics from the original post alone. When we include all social relationship information in the prompt, the similarity between human comments and LLM-generated comments decreases. This inconsistency may occur because LLMs did not include social context information as part of their training data. Together these results demonstrate the ability of LLMs to comprehend semantics from the original post and respond similarly to human comments, but also highlights their limitations in generalizing personalized comments through prompting alone.