An AI-Powered Research Assistant in the Lab: A Practical Guide for Text Analysis Through Iterative Collaboration with LLMs

TL;DR

This paper provides a practical tutorial on developing and applying taxonomies for text analysis using large language models, emphasizing an iterative, collaborative approach to improve efficiency and reliability.

Contribution

It introduces a step-by-step method for researchers to leverage LLMs in creating and refining taxonomies for analyzing unstructured data.

Findings

High intercoder reliability achieved in dataset categorization

Effective iterative process for taxonomy development demonstrated

Guidelines for prompt design and taxonomy refinement provided

Abstract

Analyzing texts such as open-ended responses, headlines, or social media posts is a time- and labor-intensive process highly susceptible to bias. LLMs are promising tools for text analysis, using either a predefined (top-down) or a data-driven (bottom-up) taxonomy, without sacrificing quality. Here we present a step-by-step tutorial to efficiently develop, test, and apply taxonomies for analyzing unstructured data through an iterative and collaborative process between researchers and LLMs. Using personal goals provided by participants as an example, we demonstrate how to write prompts to review datasets and generate a taxonomy of life domains, evaluate and refine the taxonomy through prompt and direct modifications, test the taxonomy and assess intercoder agreements, and apply the taxonomy to categorize an entire dataset with high intercoder reliability. We discuss the possibilities and limitations of using LLMs for text analysis.