Understanding the Challenges and Opportunities of Generative AI Apps: An Empirical Study

TL;DR

This study analyzes over a million reviews of Gen-AI apps to understand user perceptions, identify key topics, and uncover opportunities and challenges for future development.

Contribution

It introduces the SARA framework for large-scale review analysis using prompt-based LLMs and validates its effectiveness with high accuracy.

Findings

Identified top user concerns and interests, such as AI performance and emotional connection.

Uncovered opportunities in accessibility, wellbeing, and creative collaboration.

Highlighted challenges like managing expectations and balancing moderation.

Abstract

The release of ChatGPT in 2022 triggered a rapid surge in generative artificial intelligence mobile apps (Gen-AI apps). Despite widespread adoption, little is known about how end users perceive and evaluate these Gen-AI functionalities. We conduct a user-centered analysis of 1,035,342 reviews from 171 Gen-AI apps from the Google Play Store. We propose SARA (Selection, Acquisition, Refinement, and Analysis), a four-phase framework that leverages prompt-based LLMs for large-scale review analysis. We validate the reliability of LLM-based topic extraction and assignment using 4,353 manually evaluated reviews, achieving 91% accuracy with five-shot prompting and filtering of non-informative reviews. We identify the top ten topics (e.g., AI Performance and Emotional Connection) and perform a cross-platform comparison with Apple App Store reviews. Through qualitative analysis of 762 reviews, we uncover three opportunities (AI for Accessibility and Wellbeing, AI as a Collaborative Creative Tool, and AI Versatility) and three challenges (Managing User Expectations and AI Limitations, Balancing Content Moderation and Creative Freedom, and Strategic Integration of Gen-AI Features). Finally, we analyze temporal trends, revealing how user concerns shift as users mature. Our findings enable researchers and developers to better leverage the capabilities of Gen-AI apps and address potential challenges.