Can machines perform a qualitative data analysis? Reading the debate with Alan Turing

TL;DR

This paper reexamines the debate on using Large Language Models for qualitative data analysis, emphasizing empirical evaluation of their performance compared to human analysts, inspired by Turing's foundational ideas.

Contribution

It shifts the focus from whether machines can perform qualitative analysis to whether LLMs can produce analyses comparable to humans, using Turing's framework.

Findings

Empirical evidence shows LLMs can produce analysis similar to human analysts.

The debate should focus on the quality of analysis produced by LLMs.

Critical reflections challenge the current dismissive stance on LLMs in qualitative research.

Abstract

This paper reflects on the literature that rejects the use of Large Language Models (LLMs) in qualitative data analysis. It illustrates through empirical evidence as well as critical reflections why the current critical debate is focusing on the wrong problems. The paper proposes that the focus of researching the use of the LLMs for qualitative analysis is not the method per se, but rather the empirical investigation of an artificial system performing an analysis. The paper builds on the seminal work of Alan Turing and reads the current debate using key ideas from Turing "Computing Machinery and Intelligence". This paper therefore reframes the debate on qualitative analysis with LLMs and states that rather than asking whether machines can perform qualitative analysis in principle, we should ask whether with LLMs we can produce analyses that are sufficiently comparable to human analysts. In the final part the contrary views to performing qualitative analysis with LLMs are analysed using the same writing and rhetorical style that Turing used in his seminal work, to discuss the contrary views to the main question.