Soft Measures for Extracting Causal Collective Intelligence

TL;DR

This paper explores using large language models and novel graph-based similarity measures to automate the extraction of fuzzy cognitive maps from text, aiming to improve modeling of collective intelligence.

Contribution

It introduces new soft similarity measures and evaluates their effectiveness in extracting FCMs, highlighting the potential and limitations of current NLP approaches.

Findings

Positive correlation between measures and human judgments

Fine-tuning LLMs improves FCM extraction performance

Existing measures still struggle with FCM nuances

Abstract

Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.