Constructing Reliable Social Networks from Conversational Data: An Ensemble Prompt Engineering Approach with Uncertainty Quantification

TL;DR

This paper introduces an ensemble prompt engineering pipeline with uncertainty quantification for constructing reliable social networks from conversational data, enabling scalable and systematic analysis of interaction structures.

Contribution

It presents a novel ensemble LLM approach with uncertainty measures for automated, reliable social network construction from unstructured conversational data.

Findings

Effective classification of utterances using ensemble LLMs with majority voting.

Uncertainty quantification supports human-in-the-loop review of ambiguous cases.

Application to classroom data reveals meaningful social network insights.

Abstract

Conversational data are central to the study of interaction dynamics and social structures across psychological research. However, constructing structured social networks from unstructured conversational data remains a major methodological challenge. This study presents a pipeline for reliable network construction using prompt engineering. We employ an ensemble of multiple Large Language Models (LLMs) with majority voting to automate utterance classification, overcoming the scalability limitations of manual coding and the generalizability constraints of supervised deep learning. Classification reliability is assessed through an uncertainty quantification framework based on Shannon entropy, which supports systematic human-in-the-loop review of ambiguous cases. The classified utterances are used to construct directed interaction networks for subsequent analysis. We demonstrate the utility of this approach through two illustrative applications to classroom interaction data: network centrality analysis to characterize participant roles, and network mediation analysis using the additive and multiplicative effects network (AMEN) model to examine how interaction structures mediate the relationship between gender and mathematics performance. This pipeline provides a scalable foundation for automated network construction from conversational data across diverse research contexts.