How Users Perceive Mixed-Initiative AI: Attitudes Toward Assistance in Problem Solving

TL;DR

This study examines how different AI assistance delivery modes in problem-solving tasks influence user perceptions, finding that pre-scheduled help leads to more positive attitudes regardless of task success.

Contribution

It introduces a comparison of on-demand versus pre-scheduled AI assistance modes in a realistic puzzle task, highlighting their impact on user perception.

Findings

Pre-scheduled assistance improves user perception of AI.

Task performance was similar across assistance modes.

User attitudes are influenced by assistance delivery method.

Abstract

In mixed-initiative systems, the mode of AI assistance delivery can be as consequential as the assistance itself. We investigated two assistance delivery modes: on-demand help (users request via Button) and pre-scheduled help (assistance delivered at user-selected intervals, with user actions resetting the Timer). To evaluate these modes, we selected Rush Hour puzzles as the human-AI collaborative task because they capture elements of real-world problem solving such as analysis, resource management, and decision-making under constraints. To enhance ecological validity, we imposed monetary costs for both time and AI assistance, simulating scenarios where people must balance implicit or explicit trade-offs such as time pressure, financial limitations, or opportunity costs. Although task performance was comparable across modes, participants who used the pre-scheduled (Timer) mode reported more positive perceptions of the AI, even when their ending budget was low. This suggests that assistance delivery mode can shape user experience independent of task outcomes, indicating that human-AI systems may need to consider how AI assistance is delivered alongside improving task performance.