Expanding Horizons in HCI Research Through LLM-Driven Qualitative Analysis

TL;DR

This paper introduces a novel LLM-based qualitative analysis method for HCI research, demonstrating its effectiveness and potential to provide unique insights compared to traditional approaches.

Contribution

It presents a new LLM-driven qualitative analysis framework and a quantitative evaluation method using SBART cosine similarity in HCI research.

Findings

LLMs match traditional analysis efficacy

LLMs provide unique insights

New dataset and benchmark for LLM evaluation

Abstract

How would research be like if we still needed to "send" papers typed with a typewriter? Our life and research environment have continually evolved, often accompanied by controversial opinions about new methodologies. In this paper, we embrace this change by introducing a new approach to qualitative analysis in HCI using Large Language Models (LLMs). We detail a method that uses LLMs for qualitative data analysis and present a quantitative framework using SBART cosine similarity for performance evaluation. Our findings indicate that LLMs not only match the efficacy of traditional analysis methods but also offer unique insights. Through a novel dataset and benchmark, we explore LLMs' characteristics in HCI research, suggesting potential avenues for further exploration and application in the field.