LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments

TL;DR

This study evaluates the effectiveness of large language models in automating the annotation of security-specific comments in human experiments, finding limited success in replacing human annotators.

Contribution

It provides an empirical assessment of LLMs for security comment annotation, highlighting their current limitations and the need for further research.

Findings

LLMs show limited reliability in security comment annotation.

Detailed code descriptions improve LLM performance somewhat.

Current LLMs cannot reliably replace human annotators for this task.

Abstract

[Background:] Thematic analysis of free-text justifications in human experiments provides significant qualitative insights. Yet, it is costly because reliable annotations require multiple domain experts. Large language models (LLMs) seem ideal candidates to replace human annotators. [Problem:] Coding security-specific aspects (code identifiers mentioned, lines-of-code mentioned, security keywords mentioned) may require deeper contextual understanding than sentiment classification. [Objective:] Explore whether LLMs can act as automated annotators for technical security comments by human subjects. [Method:] We prompt four top-performing LLMs on LiveBench to detect nine security-relevant codes in free-text comments by human subjects analyzing vulnerable code snippets. Outputs are compared to human annotators using Cohen's Kappa (chance-corrected accuracy). We test different prompts mimicking annotation best practices, including emerging codes, detailed codebooks with examples, and conflicting examples. [Negative Results:] We observed marked improvements only when using detailed code descriptions; however, these improvements are not uniform across codes and are insufficient to reliably replace a human annotator. [Limitations:] Additional studies with more LLMs and annotation tasks are needed.