Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT for Mining Insights at Scale

TL;DR

This study evaluates ChatGPT's ability to extract and organize research challenges from a large HCI corpus, demonstrating its effectiveness and cost-efficiency for large-scale qualitative analysis.

Contribution

It introduces a two-step LLM-based method for mining research challenges from scholarly texts, validated with high agreement and visualized for insights.

Findings

High accuracy in challenge extraction (κ = 0.97)

Cost-effective at approximately US$50 for the entire corpus

Identified gaps in HCI research areas like human-AI collaboration

Abstract

Large language models (LLMs) are increasingly used for analytical tasks, yet their effectiveness in real-world applications remains underexamined, partly due to the opacity of proprietary models. We evaluate ChatGPT (GPT-3.5 and GPT-4) on the practical task of extracting research challenges from a large scholarly corpus in Human-Computer Interaction (HCI). Using a two-step approach, we first apply GPT-3.5 to extract candidate challenges from the 879 papers in the 2023 ACM CHI Conference proceedings, then use GPT-4 to select the most relevant challenges per paper. This process yielded 4,392 research challenges across 113 topics, which we organized through topic modeling and present in an interactive visualization. We compare the identified challenges with previously established HCI grand challenges and the United Nations Sustainable Development Goals, finding both strong alignment in areas such as ethics and accessibility, and gaps in areas such as human-AI collaboration. A task-specific evaluation with human raters confirmed near-perfect agreement that the extracted statements represent plausible research challenges (\k{appa} = 0.97). The two-step approach proved cost-effective at approximately US$50 for the full corpus, suggesting that LLMs offer a practical means for qualitative text analysis at scale, particularly for prototyping research ideas and examining corpora from multiple analytical perspectives.