LLMAID: Identifying AI Capabilities in Android Apps with LLMs

TL;DR

This paper introduces LLMAID, a novel LLM-based framework for automatically identifying and analyzing AI capabilities in Android apps, significantly outperforming previous rule-based methods in accuracy and comprehensiveness.

Contribution

The paper presents LLMAID, a new LLM-driven approach that automates AI capability detection in mobile apps, overcoming limitations of manual and rule-based methods.

Findings

LLMAID detects 242% more AI apps than previous methods.

Achieves over 90% precision and recall in AI component detection.

Most AI functionalities are in computer vision, especially object detection.

Abstract

Recent advancements in artificial intelligence (AI) and its widespread integration into mobile software applications have received significant attention, highlighting the growing prominence of AI capabilities in modern software systems. However, the inherent hallucination and reliability issues of AI continue to raise persistent concerns. Consequently, application users and regulators increasingly ask critical questions such as: Does the application incorporate AI capabilities? and What specific types of AI functionalities are embedded? Preliminary efforts have been made to identify AI capabilities in mobile software; however, existing approaches mainly rely on manual inspection and rule-based heuristics. These methods are not only costly and time-consuming but also struggle to adapt advanced AI techniques. To address the limitations of existing methods, we propose LLMAID (Large Language Model for AI Discovery). LLMAID includes four main tasks: (1) candidate extraction, (2) knowledge base interaction, (3) AI capability analysis and detection, and (4) AI service summarization. We apply LLMAID to a dataset of 4,201 Android applications and demonstrate that it identifies 242% more real-world AI apps than state-of-the-art rule-based approaches. Our experiments show that LLM4AID achieves high precision and recall, both exceeding 90%, in detecting AI-related components. Additionally, a user study indicates that developers find the AI service summaries generated by LLMAID to be more informative and preferable to the original app descriptions. Finally, we leverage LLMAID to perform an empirical analysis of AI capabilities across Android apps. The results reveal a strong concentration of AI functionality in computer vision (54.80%), with object detection emerging as the most common task (25.19%).