Thinking Assistants: LLM-Based Conversational Assistants that Help Users Think By Asking rather than Answering

TL;DR

This paper introduces 'Thinking Assistants', LLM-based conversational agents designed to enhance users' decision-making by asking reflective questions rather than providing direct answers, demonstrating improved guidance in complex tasks.

Contribution

The paper proposes a new genre of LLM-based assistants focused on prompting reflection, supported by empirical studies showing their effectiveness over traditional question-answering agents.

Findings

Thinking Assistants better guide decision-making than traditional agents.

Participants responded positively to about 65% of the assistant's responses.

Longitudinal deployment shows sustained user engagement and perceived helpfulness.

Abstract

Many AI systems focus solely on providing solutions or explaining outcomes. However, complex tasks like research and strategic thinking often benefit from a more comprehensive approach to augmenting the thinking process rather than passively getting information. We introduce the concept of "Thinking Assistant", a new genre of assistants that help users improve decision-making with a combination of asking reflection questions based on expert knowledge. Through our lab study (N=80), these Large Language Model (LLM) based Thinking Assistants were better able to guide users to make important decisions, compared with conversational agents that only asked questions, provided advice, or neither. Based on the results, we develop a Thinking Assistant in academic career development, determining research trajectory or developing one's unique research identity, which requires deliberation, reflection and experts' advice accordingly. In a longitudinal deployment with 223 conversations, participants responded positively to approximately 65% of the responses. Our work proposes directions for developing more effective LLM agents. Rather than adhering to the prevailing authoritative approach of generating definitive answers, LLM agents aimed at assisting with cognitive enhancement should prioritize fostering reflection. They should initially provide responses designed to prompt thoughtful consideration through inquiring, followed by offering advice only after gaining a deeper understanding of the user's context and needs.