Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCI

TL;DR

This paper critically examines how LLM reasoning is perceived in HCI, highlighting the need for more informed and reflective engagement by developing prompts based on interdisciplinary insights.

Contribution

It offers a systematic survey of HCI literature on LLMs and introduces reflection prompts to improve understanding of LLM reasoning in HCI contexts.

Findings

HCI often perceives LLM reasoning as an application object.

Current views oversimplify LLM reasoning methodologies.

Reflection prompts can foster more informed engagement with LLM reasoning.

Abstract

Reasoning is a distinctive human-like characteristic attributed to LLMs in HCI due to their ability to simulate various human-level tasks. However, this work argues that the reasoning behavior of LLMs in HCI is often decontextualized from the underlying mechanics and subjective decisions that condition the emergence and human interpretation of this behavior. Through a systematic survey of 258 CHI papers from 2020-2025 on LLMs, we discuss how HCI hardly perceives LLM reasoning as a product of sociotechnical orchestration and often references it as an object of application. We argue that such abstraction leads to oversimplification of reasoning methodologies from NLP/ML and results in a distortion of LLMs' empirically studied capabilities and (un)known limitations. Finally, drawing on literature from both NLP/ML and HCI, as a constructive step forward, we develop reflection prompts to support HCI practitioners engage with LLM reasoning in an informed and reflective way.