Interaction-Augmented Instruction: Modeling the Synergy of Prompts and Interactions in Human-GenAI Collaboration

TL;DR

This paper introduces the Interaction-Augmented Instruction (IAI) model, a formal framework that captures how combining prompts and interactions improves human-GenAI collaboration, enabling better design and innovation.

Contribution

The paper presents a novel formal model, IAI, that systematically characterizes the synergy of prompts and interactions in human-GenAI communication.

Findings

Developed the IAI model as an entity-relation graph.

Identified twelve atomic interaction paradigms from existing tools.

Demonstrated the model's utility in design, comparison, and innovation scenarios.

Abstract

Text prompt is the most common way for human-generative AI (GenAI) communication. Though convenient, it is challenging to convey fine-grained and referential intent. One promising solution is to combine text prompts with precise GUI interactions, like brushing and clicking. However, there lacks a formal model to capture synergistic designs between prompts and interactions, hindering their comparison and innovation. To fill this gap, via an iterative and deductive process, we develop the Interaction-Augmented Instruction (IAI) model, a compact entity-relation graph formalizing how the combination of interactions and text prompts enhances human-GenAI communication. With the model, we distill twelve recurring and composable atomic interaction paradigms from prior tools, verifying our model's capability to facilitate systematic design characterization and comparison. Four usage scenarios further demonstrate the model's utility in applying, refining, and innovating these paradigms. These results illustrate the IAI model's descriptive, discriminative, and generative power for shaping future GenAI systems.