Leveraging LLM-based agents for social science research: insights from citation network simulations

TL;DR

This paper introduces the CiteAgent framework using LLMs to simulate citation networks, enabling new research paradigms in social science and providing insights into citation phenomena and scientific behaviors.

Contribution

It presents a novel LLM-based simulation framework for citation networks and establishes new research paradigms for social science studies using LLMs.

Findings

CiteAgent captures real-world citation network phenomena

LLM-based paradigms enable validation of social science theories

Simulation results offer insights into academic behaviors

Abstract

The emergence of Large Language Models (LLMs) demonstrates their potential to encapsulate the logic and patterns inherent in human behavior simulation by leveraging extensive web data pre-training. However, the boundaries of LLM capabilities in social simulation remain unclear. To further explore the social attributes of LLMs, we introduce the CiteAgent framework, designed to generate citation networks based on human-behavior simulation with LLM-based agents. CiteAgent successfully captures predominant phenomena in real-world citation networks, including power-law distribution, citational distortion, and shrinking diameter. Building on this realistic simulation, we establish two LLM-based research paradigms in social science: LLM-SE (LLM-based Survey Experiment) and LLM-LE (LLM-based Laboratory Experiment). These paradigms facilitate rigorous analyses of citation network phenomena, allowing us to validate and challenge existing theories. Additionally, we extend the research scope of traditional science of science studies through idealized social experiments, with the simulation experiment results providing valuable insights for real-world academic environments. Our work demonstrates the potential of LLMs for advancing science of science research in social science.