Towards Real Smart Apps: Investigating Human-AI Interactions in Smartphone On-Device AI Apps

TL;DR

This paper investigates human-AI interactions in smartphone on-device AI apps, analyzing 176 apps to identify interaction patterns and inform better design, supported by a user study validating the findings.

Contribution

First empirical study exploring user-AI interaction in mobile apps, identifying key interaction patterns and implementing a search-enabled gallery for improved design guidance.

Findings

Identified 255 AI features across 176 apps

Summarized 759 implementations into three primary interaction patterns

User study confirms the usefulness of the identified patterns

Abstract

With the emergence of deep learning techniques, smartphone apps are now embedded on-device AI features for enabling advanced tasks like speech translation, to attract users and increase market competitiveness. A good interaction design is important to make an AI feature usable and understandable. However, AI features have their unique challenges like sensitiveness to the input, dynamic behaviours and output uncertainty. Existing guidelines and tools either do not cover AI features or consider mobile apps which are confirmed by our informal interview with professional designers. To address these issues, we conducted the first empirical study to explore user-AI-interaction in mobile apps. We aim to understand the status of on-device AI usage by investigating 176 AI apps from 62,822 apps. We identified 255 AI features and summarised 759 implementations into three primary interaction pattern types. We further implemented our findings into a multi-faceted search-enabled gallery. The results of the user study demonstrate the usefulness of our findings.