In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker

TL;DR

MobileMaker is a platform that allows designers to quickly create and test multimodal AI prototypes directly on mobile devices, enabling real-time user feedback and iterative improvements in natural settings.

Contribution

The paper introduces MobileMaker, a novel in situ prototyping tool for mobile AI features that supports natural language revisions and real-world testing, filling a gap in mobile AI development workflows.

Findings

MobileMaker prototypes revealed more edge cases and contextual signals missed by AI.

Users felt more engaged and fulfilled when making on-device revisions.

Revisions constrained feedback to more actionable changes.

Abstract

Recent advances in multimodal large language models (LLMs) have made it easier to rapidly prototype AI-powered features, especially for mobile use cases. However, gathering early, mobile-situated user feedback on these AI prototypes remains challenging. The broad scope and flexibility of LLMs means that, for a given use-case-specific prototype, there is a crucial need to understand the wide range of in-the-wild input users are likely to provide and their in-context expectations for the AI's behavior. To explore the concept of in situ AI prototyping and testing, we created MobileMaker: a platform that enables designers to rapidly create and test mobile AI prototypes directly on devices. This tool also enables testers to make on-device, in-the-field revisions of prototypes using natural language. In an exploratory study with 16 participants, we explored how user feedback on prototypes created with MobileMaker compares to that of existing prototyping tools (e.g., Figma, prompt editors). Our findings suggest that MobileMaker prototypes enabled more serendipitous discovery of: model input edge cases, discrepancies between AI's and user's in-context interpretation of the task, and contextual signals missed by the AI. Furthermore, we learned that while the ability to make in-the-wild revisions led users to feel more fulfilled as active participants in the design process, it might also constrain their feedback to the subset of changes perceived as more actionable or implementable by the prototyping tool.