Apprentices to Research Assistants: Advancing Research with Large Language Models

TL;DR

This paper reviews and experiments with Large Language Models as research tools, highlighting their benefits, challenges, and strategies for effective and ethical integration into research workflows.

Contribution

It provides empirical insights into LLMs' capabilities and limitations in research, proposing strategies for prompt tuning and bias mitigation.

Findings

LLMs can assist in qualitative research analysis

Prompt optimization improves LLM performance

Bias and subjectivity remain significant challenges

Abstract

Large Language Models (LLMs) have emerged as powerful tools in various research domains. This article examines their potential through a literature review and firsthand experimentation. While LLMs offer benefits like cost-effectiveness and efficiency, challenges such as prompt tuning, biases, and subjectivity must be addressed. The study presents insights from experiments utilizing LLMs for qualitative analysis, highlighting successes and limitations. Additionally, it discusses strategies for mitigating challenges, such as prompt optimization techniques and leveraging human expertise. This study aligns with the 'LLMs as Research Tools' workshop's focus on integrating LLMs into HCI data work critically and ethically. By addressing both opportunities and challenges, our work contributes to the ongoing dialogue on their responsible application in research.