Depth and Autonomy: A Framework for Evaluating LLM Applications in Social Science Research

TL;DR

This paper proposes a framework to evaluate and guide the use of large language models in social science research by balancing interpretive depth and autonomy to enhance transparency and reliability.

Contribution

It introduces a novel two-dimensional framework for classifying LLM applications in qualitative social science research and offers practical design recommendations.

Findings

Classifies LLM applications along interpretive depth and autonomy.

Recommends decomposing tasks and controlling model autonomy.

Supports improved transparency and reliability in research use.

Abstract

Large language models (LLMs) are increasingly utilized by researchers across a wide range of domains, and qualitative social science is no exception; however, this adoption faces persistent challenges, including interpretive bias, low reliability, and weak auditability. We introduce a framework that situates LLM usage along two dimensions, interpretive depth and autonomy, thereby offering a straightforward way to classify LLM applications in qualitative research and to derive practical design recommendations. We present the state of the literature with respect to these two dimensions, based on all published social science papers available on Web of Science that use LLMs as a tool and not strictly as the subject of study. Rather than granting models expansive freedom, our approach encourages researchers to decompose tasks into manageable segments, much as they would when delegating work to capable undergraduate research assistants. By maintaining low levels of autonomy and selectively increasing interpretive depth only where warranted and under supervision, one can plausibly reap the benefits of LLMs while preserving transparency and reliability.