Conversational Customization of Productivity Systems: A Design Probe of Malleable AI Interfaces

TL;DR

This study explores how users interact with a conversationally customizable email system, revealing that they adapt existing patterns, manage risks, and refine features over time, highlighting the importance of support for iterative customization.

Contribution

The paper introduces a design probe for studying natural language-based customization in productivity tools, providing insights into user behavior and system design implications.

Findings

Users adapt existing patterns rather than creating entirely new functionalities.

Customization shifts user engagement from fixed interfaces to a flexible data layer.

Users manage risks through ongoing oversight and iterative refinement.

Abstract

Customization has long been a central goal in interactive systems, yet prior work shows that end-user tailoring occurs infrequently and is often confined to initial setup or moments of breakdown. Recent advances in generative AI suggest that highly malleable systems-where users can modify system behavior through natural language-are now technically feasible. However, it remains unclear how such malleability is used in practice: What kinds of customizations do users create, when do they choose to customize, and how do these modifications shape their experience of everyday tools? We present a design probe that uses a conversationally customizable email system as an instrument to study how users create and refine functionality within everyday tools. The system allows users to iteratively modify their inbox by restructuring categories, introducing interface elements, and authoring new workflow behaviors directly through natural language interaction. We study how participants create, refine, and use these features over several days within their own email workflows. We find that users' customizations are often grounded in existing patterns, which they adapt and specialize to fit their needs, rather than generating entirely novel functionality. Malleability changes how users engage with their inbox, shifting it from a fixed interface to a flexible data layer shaped through user-authored features. At the same time, customization introduces new forms of risk, including mis-specified behavior, unintended filtering, and uncertainty around outcomes, which users manage through ongoing oversight and refinement. These findings highlight how conversational customization becomes embedded within everyday interaction, and point toward the need for systems that support iterative refinement, visibility into behavior, and safe experimentation as users shape their own tools.