GPT-ology, Computational Models, Silicon Sampling: How should we think about LLMs in Cognitive Science?

TL;DR

This paper reviews emerging research paradigms involving Large Language Models in cognitive science, discussing their claims, challenges, and issues like transparency, reproducibility, and open vs closed models.

Contribution

It provides a comprehensive overview of new paradigms for using LLMs in cognitive science and discusses key challenges to scientific inference.

Findings

Review of GPT-ology, LLMs-as-models, and silicon sampling paradigms

Identification of challenges like transparency, reproducibility, and open-source issues

Discussion of the need for conventions in task hyperparameters

Abstract

Large Language Models have taken the cognitive science world by storm. It is perhaps timely now to take stock of the various research paradigms that have been used to make scientific inferences about ``cognition" in these models or about human cognition. We review several emerging research paradigms -- GPT-ology, LLMs-as-computational-models, and ``silicon sampling" -- and review recent papers that have used LLMs under these paradigms. In doing so, we discuss their claims as well as challenges to scientific inference under these various paradigms. We highlight several outstanding issues about LLMs that have to be addressed to push our science forward: closed-source vs open-sourced models; (the lack of visibility of) training data; and reproducibility in LLM research, including forming conventions on new task ``hyperparameters" like instructions and prompts.