Evaluating LLM-Based Mobile App Recommendations: An Empirical Study

TL;DR

This empirical study investigates how large language models recommend mobile apps, analyzing their ranking criteria, consistency, and responsiveness to instructions, revealing fragmented criteria and complex reasoning dynamics.

Contribution

The paper introduces a taxonomy of ranking criteria, a systematic evaluation framework, and a replication package for analyzing LLM-based app recommendations.

Findings

LLMs rely on fragmented ranking criteria only partially aligned with ASO metrics.

Top-ranked apps are consistent across runs, but variability increases with depth and specificity.

LLMs show varying sensitivity to explicit instructions, indicating complex reasoning dynamics.

Abstract

Large Language Models (LLMs) are increasingly used to recommend mobile applications through natural language prompts, offering a flexible alternative to keyword-based app store search. Yet, the reasoning behind these recommendations remains opaque, raising questions about their consistency, explainability, and alignment with traditional App Store Optimization (ASO) metrics. In this paper, we present an empirical analysis of how widely-used general purpose LLMs generate, justify, and rank mobile app recommendations. Our contributions are: (i) a taxonomy of 16 generalizable ranking criteria elicited from LLM outputs; (ii) a systematic evaluation framework to analyse recommendation consistency and responsiveness to explicit ranking instructions; and (iii) a replication package to support reproducibility and future research on AI-based recommendation systems. Our findings reveal that LLMs rely on a broad yet fragmented set of ranking criteria, only partially aligned with standard ASO metrics. While top-ranked apps tend to be consistent across runs, variability increases with ranking depth and search specificity. LLMs exhibit varying sensitivity to explicit ranking instructions - ranging from substantial adaptations to near-identical outputs - highlighting their complex reasoning dynamics in conversational app discovery. Our results aim to support end-users, app developers, and recommender-systems researchers in navigating the emerging landscape of conversational app discovery.