From Verification Burden to Trusted Collaboration: Design Goals for LLM-Assisted Literature Reviews

TL;DR

This paper investigates how researchers use Large Language Models in literature reviews, identifying trust and verification challenges, and proposes a framework with design goals to improve collaboration and reliability in AI-assisted scientific writing.

Contribution

It introduces a set of design goals and a high-level framework to enhance trust, verification, and usability of LLMs in literature review processes based on user study insights.

Findings

Identified trust and verification as key challenges in LLM-assisted reviews.

Proposed a framework with visualization and verification features to address these challenges.

Designed a system that aligns human feedback with AI explanations for better collaboration.

Abstract

Large Language Models (LLMs) are increasingly embedded in academic writing practices. Although numerous studies have explored how researchers employ these tools for scientific writing, their concrete implementation, limitations, and design challenges within the literature review process remain underexplored. In this paper, we report a user study with researchers across multiple disciplines to characterize current practices, benefits, and \textit{pain points} in using LLMs to investigate related work. We identified three recurring gaps: (i) lack of trust in outputs, (ii) persistent verification burden, and (iii) requiring multiple tools. This motivates our proposal of six design goals and a high-level framework that operationalizes them through improved related papers visualization, verification at every step, and human-feedback alignment with generation-guided explanations. Overall, by grounding our work in the practical, day-to-day needs of researchers, we designed a framework that addresses these limitations and models real-world LLM-assisted writing, advancing trust through verifiable actions and fostering practical collaboration between researchers and AI systems.