On the (In)Security of LLM App Stores

TL;DR

This paper investigates the security risks of large language model app stores by analyzing over 786,000 apps, revealing significant threats like malicious content, privacy violations, and potential for misuse, and proposes a framework for risk identification.

Contribution

It introduces a three-layer concern framework for assessing LLM app security risks and provides large-scale empirical analysis using static/dynamic methods and toxic word detection.

Findings

15,146 apps had misleading descriptions

1,366 apps collected sensitive data illegally

15,996 apps generated harmful content

Abstract

LLM app stores have seen rapid growth, leading to the proliferation of numerous custom LLM apps. However, this expansion raises security concerns. In this study, we propose a three-layer concern framework to identify the potential security risks of LLM apps, i.e., LLM apps with abusive potential, LLM apps with malicious intent, and LLM apps with exploitable vulnerabilities. Over five months, we collected 786,036 LLM apps from six major app stores: GPT Store, FlowGPT, Poe, Coze, Cici, and Character.AI. Our research integrates static and dynamic analysis, the development of a large-scale toxic word dictionary (i.e., ToxicDict) comprising over 31,783 entries, and automated monitoring tools to identify and mitigate threats. We uncovered that 15,146 apps had misleading descriptions, 1,366 collected sensitive personal information against their privacy policies, and 15,996 generated harmful content such as hate speech, self-harm, extremism, etc. Additionally, we evaluated the potential for LLM apps to facilitate malicious activities, finding that 616 apps could be used for malware generation, phishing, etc. Our findings highlight the urgent need for robust regulatory frameworks and enhanced enforcement mechanisms.