The Shiny Scary Future of Automated Research Synthesis in HCI

TL;DR

This paper discusses the potential and risks of using automated tools like LLMs in research synthesis within HCI, emphasizing the need for caution to preserve human judgment.

Contribution

It highlights the opportunities and challenges of integrating automation in research synthesis, advocating for careful application to maintain research quality.

Findings

Automation can save time in research synthesis

Caution is needed to preserve human judgment

Automated tools may impact research quality

Abstract

Automation and semi-automation through computational tools like LLMs are also making their way to deployment in research synthesis and secondary research, such as systematic reviews. In some steps of research synthesis, this has the opportunity to provide substantial benefits by saving time that previously was spent on repetitive tasks. The screening stages in particular may benefit from carefully vetted computational support. However, this position paper argues for additional caution when bringing in such tools to the analysis and synthesis phases, where human judgement and expertise should be paramount throughout the process.