Not Too Short, Not Too Long: How LLM Response Length Shapes People's Critical Thinking in Error Detection

TL;DR

This study investigates how the length of LLM responses influences users' ability to critically evaluate reasoning, finding that response correctness and length interact to affect accuracy in critical thinking tasks.

Contribution

It provides empirical evidence on the moderating role of response length in LLM-assisted critical thinking, highlighting design implications for AI explanations.

Findings

Participants more accurate with correct LLM explanations.

Medium-length explanations improve accuracy when LLM is incorrect.

Accuracy remains high across lengths when LLM explanations are correct.

Abstract

Large language models (LLMs) have become common decision-support tools across educational and professional contexts, raising questions about how their outputs shape human critical thinking. Prior work suggests that the amount of AI assistance can influence cognitive engagement, yet little is known about how specific properties of LLM outputs (e.g., response length) impacts users' critical evaluation of information. In this study, we examine whether the length of LLM responses shapes users' accuracy in evaluating LLM-generated reasoning on critical thinking tasks, particularly in interaction with the correctness of the LLM's reasoning. To begin evaluating this, we conducted a within-subjects experiment with 24 participants who completed 15 modified Watson--Glaser critical thinking items, each accompanied by an LLM-generated explanation that varied in length and correctness. Mixed-effects logistic regression revealed a strong and statistically reliable effect of LLM output correctness on participant accuracy, with participants more likely to answer correctly when the LLM's explanation was correct. Response length appeared to moderated this effect: when the LLM output was incorrect, medium-length explanations were associated with higher participant accuracy than either shorter or longer explanations, whereas accuracy remained high across lengths when the LLM output was correct. Together, these findings suggest that response length alone may be insufficient to support critical thinking, and that how reasoning is presented-including a potential advantage of mid-length explanations under some conditions-points to design opportunities for LLM-based decision-support systems that emphasize transparent reasoning and calibrated expressions of certainty.