Compass vs Railway Tracks: Unpacking User Mental Models for Communicating Long-Horizon Work to Humans vs. AI

TL;DR

This study explores how users communicate long-horizon tasks differently to humans versus AI, revealing a preference for high-level intent with humans and detailed instructions with AI, and proposes design implications for future AI collaboration.

Contribution

It uncovers the contrasting mental models users have for communicating with humans and AI in long-term tasks and suggests design strategies for more effective AI-human collaboration.

Findings

Participants treat human delegation as a 'compass' with high-level intent.

Communication with AI resembles laying 'railway tracks' with exhaustive instructions.

Users desire a hybrid AI that combines efficiency with human-like critical thinking.

Abstract

As AI systems grow increasingly capable of operating for hours or days at a time, users' prompts are transforming into elaborate specifications for the AI to autonomously work on. While prompting for bounded, single-turn tasks has been extensively studied, less is known about how people communicate specifications for long-horizon tasks. We conducted a qualitative study in which 16 professionals drafted specifications for both a human colleague and an AI, revealing a core divergence: participants treated human delegation as a "compass", offering high-level intent to encourage flexible exploration. In contrast, communication with AI resembled painstakingly laying down "railway tracks": rigid, exhaustive instructions to minimize ambiguity and deviation. This reflected a perception that current AI struggles to infer intent, prioritize, and make judgments on its own. When envisioning an ideal AI collaborator, users desired a hybrid: a collaborator blending AI's efficiency and large context window with the critical thinking and agency of a human colleague. We discuss design implications for future AI systems, proposing that they align on outcomes through generated rough drafts, verify feasibility via end-to-end "test runs," and monitor execution through intelligent check-ins -- ultimately transforming AI from a passive instruction-follower into a reliable collaborator for ambiguous, long-horizon tasks.