AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools

TL;DR

AI-Instruments transforms prompt-based AI interactions into manipulable interface objects, enabling more intuitive, reflective, and iterative design workflows by embodying prompts as reusable instruments grounded in user intent.

Contribution

This paper introduces AI-Instruments, a novel approach that reifies prompts as interactive tools, incorporating reflection and grounding principles, and leverages LLMs to generate and refine instruments for creative AI tasks.

Findings

Enhanced exploration and refinement of ambiguous user intents.

Improved control and iteration in AI-assisted design workflows.

Qualitative insights from twelve participants demonstrate effectiveness.

Abstract

Chat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody "prompts" as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manipulation instruments; (2) Reflection of multiple interpretations of ambiguous user-intents (Reflection-in-intent) as well as the range of AI-model responses (Reflection-in-response) to inform design "moves" towards a desired result; and (3) Grounding to instantiate an instrument from an example, result, or extrapolation directly from another instrument. Further, AI-Instruments leverage LLM's to suggest, vary, and refine new instruments, enabling a system that goes beyond hard-coded functionality by generating its own instrumental controls from content. We demonstrate four technology probes, applied to image generation, and qualitative insights from twelve participants, showing how AI-Instruments address challenges of intent formulation, steering via direct manipulation, and non-linear iterative workflows to reflect and resolve ambiguous intents.