GenAI Is No Silver Bullet for Qualitative Research in Software Engineering

TL;DR

While GenAI offers potential support for qualitative research in software engineering, its effectiveness is limited and must be carefully tailored to specific research strategies and data types, highlighting both opportunities and pitfalls.

Contribution

This paper critically examines the use of GenAI in qualitative software engineering research, emphasizing its limitations and the need for careful adaptation.

Findings

GenAI support requires careful adaptation to specific data and strategies.

Emerging evidence shows mixed results for GenAI assistance in qualitative research.

GenAI introduces both benefits and challenges in maintaining research quality.

Abstract

Qualitative research gives rich insights into the quintessentially human aspects of software engineering as a socio-technical system. Qualitative research spans diverse strategies and methods, from interpretivist, in situ observational field studies, to deductive coding of data from mining studies. Advances in large language models and generative AI (GenAI) have prompted claims that artificial intelligence could automate qualitative analysis. Such claims are overgeneralizing from narrow successes. GenAI support must be carefully adapted to the data of interest, but also to the characteristics of a particular research strategy. In this Frontiers of SE paper, we discuss the emerging use of GenAI in relation to the broad spectrum of qualitative research in software engineering. We outline the dimensions of qualitative work in software engineering, review emerging empirical evidence for GenAI assistance, examine the pros and cons of GenAI-mediated qualitative research practices, and revisit qualitative research quality factors, in light of GenAI. Our goal is to inform researchers about the promises and pitfalls of GenAI-assisted qualitative research. We conclude with future plans to advance understanding of its use in software engineering.