Unraveling SITT: Social Influence Technique Taxonomy and Detection with LLMs

TL;DR

This paper introduces SITT, a detailed taxonomy of social influence techniques, and evaluates LLMs' ability to detect these techniques in dialogues, revealing current limitations and the need for domain-specific fine-tuning.

Contribution

The paper presents the first comprehensive taxonomy of social influence techniques and a new annotated dataset for evaluating LLMs' detection capabilities.

Findings

Claude 3.5 achieved moderate success with F1 score = 0.45

LLMs show limited performance on context-sensitive techniques

Current LLMs have difficulty detecting nuanced social influence cues

Abstract

In this work we present the Social Influence Technique Taxonomy (SITT), a comprehensive framework of 58 empirically grounded techniques organized into nine categories, designed to detect subtle forms of social influence in textual content. We also investigate the LLMs ability to identify various forms of social influence. Building on interdisciplinary foundations, we construct the SITT dataset -- a 746-dialogue corpus annotated by 11 experts in Polish and translated into English -- to evaluate the ability of LLMs to identify these techniques. Using a hierarchical multi-label classification setup, we benchmark five LLMs, including GPT-4o, Claude 3.5, Llama-3.1, Mixtral, and PLLuM. Our results show that while some models, notably Claude 3.5, achieved moderate success (F1 score = 0.45 for categories), overall performance of models remains limited, particularly for context-sensitive techniques. The findings demonstrate key limitations in current LLMs' sensitivity to nuanced linguistic cues and underscore the importance of domain-specific fine-tuning. This work contributes a novel resource and evaluation example for understanding how LLMs detect, classify, and potentially replicate strategies of social influence in natural dialogues.