TACOMORE: Leveraging the Potential of LLMs in Corpus-based Discourse Analysis with Prompt Engineering

TL;DR

TACOMORE is a prompting framework designed to enhance the performance, reproducibility, and ethicality of large language models in automated corpus-based discourse analysis, demonstrated on COVID-19 research articles.

Contribution

The paper introduces TACOMORE, a novel prompt engineering framework that improves LLMs' effectiveness in discourse analysis tasks with better reproducibility and ethical considerations.

Findings

Improved LLM performance on keyword, collocate, and concordance analysis

Enhanced reproducibility and ethicality in discourse analysis

Provides a structured prompting approach for qualitative research

Abstract

The capacity of LLMs to carry out automated qualitative analysis has been questioned by corpus linguists, and it has been argued that corpus-based discourse analysis incorporating LLMs is hindered by issues of unsatisfying performance, hallucination, and irreproducibility. Our proposed method, TACOMORE, aims to address these concerns by serving as an effective prompting framework in this domain. The framework consists of four principles, i.e., Task, Context, Model and Reproducibility, and specifies five fundamental elements of a good prompt, i.e., Role Description, Task Definition, Task Procedures, Contextual Information and Output Format. We conduct experiments on three LLMs, i.e., GPT-4o, Gemini-1.5-Pro and Gemini-1.5.Flash, and find that TACOMORE helps improve LLM performance in three representative discourse analysis tasks, i.e., the analysis of keywords, collocates and concordances, based on an open corpus of COVID-19 research articles. Our findings show the efficacy of the proposed prompting framework TACOMORE in corpus-based discourse analysis in terms of Accuracy, Ethicality, Reasoning, and Reproducibility, and provide novel insights into the application and evaluation of LLMs in automated qualitative studies.