Reporting and Reviewing LLM-Integrated Systems in HCI: Challenges and Considerations
Karla Felix Navarro, Eugene Syriani, Ian Arawjo

TL;DR
This paper explores challenges in reporting and reviewing LLM-integrated systems in HCI, highlighting trust issues, review biases, and community norm clashes, and offers guidelines for better practices.
Contribution
It provides empirical insights from interviews with authors and reviewers, identifying key challenges and proposing guidelines for reporting and reviewing LLM-integrated systems in HCI.
Findings
Reviewers apply skeptical standards to LLM papers.
Authors add technical evaluations to mitigate mistrust.
Community norms clash over contribution definitions.
Abstract
What should HCI scholars consider when reporting and reviewing papers that involve LLM-integrated systems? We interview 18 authors of LLM-integrated system papers on their authoring and reviewing experiences. We find that norms of trust-building between authors and reviewers appear to be eroded by the uncertainty of LLM behavior and hyperbolic rhetoric surrounding AI. Authors perceive that reviewers apply uniquely skeptical and inconsistent standards towards papers that report LLM-integrated systems, and mitigate mistrust by adding technical evaluations, justifying usage, and de-emphasizing LLM presence. Authors' views challenge blanket directives to report all prompts and use open models, arguing that prompt reporting is context-dependent and justifying proprietary model usage despite ethical concerns. Finally, some tensions in peer review appear to stem from clashes between the norms…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAI in Service Interactions · Ethics and Social Impacts of AI · Expert finding and Q&A systems
