Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models

TL;DR

This paper presents Interactive Reasoning, a visualization and control interface for chain-of-thought outputs in large language models, enabling user review, editing, and improved understanding of model reasoning.

Contribution

It introduces a novel interaction design that visualizes and allows editing of reasoning chains, integrating user oversight into LLM reasoning processes.

Findings

Users can identify and correct errors more quickly.

Interactive reasoning improves user understanding of model outputs.

Users can steer model responses more effectively.

Abstract

The output quality of large language models (LLMs) can be improved via "reasoning": generating segments of chain-of-thought (CoT) content to further condition the model prior to producing user-facing output. While these chains contain valuable information, they are verbose and lack explicit organization, making them tedious to review. Moreover, they lack opportunities for user feedback, such as to remove unwanted considerations, add desired ones, or clarify unclear assumptions. We introduce Interactive Reasoning, an interaction design that visualizes chain-of-thought outputs as a hierarchy of topics and enables user review and modification. We implement interactive reasoning in Hippo, a prototype for AI-assisted decision making in the face of uncertain trade-offs. In a user study with 16 participants, we find that interactive reasoning in Hippo allows users to quickly identify and interrupt erroneous generations, efficiently steer the model towards customized responses, and better understand both model reasoning and model outputs. Our work contributes to a new paradigm that incorporates user oversight into LLM reasoning processes.