Selecting Language Models for Social Science: Start Small, Start Open, and Validate

TL;DR

This paper advocates for social scientists to select language models by prioritizing openness, smaller size, and validation through reproducibility, emphasizing the importance of ex-post validation and constructing benchmarks for reliable research.

Contribution

It introduces a framework for selecting LLMs based on validity, openness, and reproducibility, emphasizing starting small and validating models through benchmarks.

Findings

Reproducibility is crucial for model selection in social science.

Starting with smaller, open models facilitates validation.

Constructing benchmarks ensures the validity of computational pipelines.

Abstract

Currently, there are thousands of large pretrained language models (LLMs) available to social scientists. How do we select among them? Using validity, reliability, reproducibility, and replicability as guides, we explore the significance of: (1) model openness, (2) model footprint, (3) training data, and (4) model architectures and fine-tuning. While ex-ante tests of validity (i.e., benchmarks) are often privileged in these discussions, we argue that social scientists cannot altogether avoid validating computational measures (ex-post). Replicability, in particular, is a more pressing guide for selecting language models. Being able to reliably replicate a particular finding that entails the use of a language model necessitates reliably reproducing a task. To this end, we propose starting with smaller, open models, and constructing delimited benchmarks to demonstrate the validity of the entire computational pipeline.