Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflow

TL;DR

This longitudinal study investigates how users familiarize themselves with and customize generative AI workflows over time, revealing sustained utility and highlighting the importance of customization for user engagement.

Contribution

The paper provides empirical evidence that perceived utility of AI workflows increases after familiarization, emphasizing the role of customization in user adaptation and system appropriation.

Findings

Perceived utility increases after user familiarization.

Customization of prompts enhances perceived benefits.

Users develop anticipation of AI outputs over time.

Abstract

Generative AI brings novel and impressive abilities to help people in everyday tasks. There are many AI workflows that solve real and complex problems by chaining AI outputs together with human interaction. Although there is an undeniable lure of AI, it is uncertain how useful generative AI workflows are after the novelty wears off. Additionally, workflows built with generative AI have the potential to be easily customized to fit users' individual needs, but do users take advantage of this? We conducted a three-week longitudinal study with 12 users to understand the familiarization and customization of generative AI tools for science communication. Our study revealed that there exists a familiarization phase, during which users were exploring the novel capabilities of the workflow and discovering which aspects they found useful. After this phase, users understood the workflow and were able to anticipate the outputs. Surprisingly, after familiarization the perceived utility of the system was rated higher than before, indicating that the perceived utility of AI is not just a novelty effect. The increase in benefits mainly comes from end-users' ability to customize prompts, and thus potentially appropriate the system to their own needs. This points to a future where generative AI systems can allow us to design for appropriation.