LLM-Powered Analysis of IoT User Reviews: Tracking and Ranking Security and Privacy Concerns

TL;DR

This paper introduces a GPT-3.5-based pipeline to automatically identify and categorize IoT user reviews expressing security and privacy concerns, revealing persistent issues and user demands across multiple device types.

Contribution

The work presents a novel automated approach using large language models to analyze IoT reviews for S&P concerns, outperforming traditional methods and uncovering new themes.

Findings

Over 97% precision and recall in identifying S&P concerns.

Detected significantly more S&P reviews than prior methods.

Persistent user concerns about surveillance and data control.

Abstract

Being able to understand the security and privacy (S&P) concerns of IoT users brings benefits to both developers and users. To learn about users' views, we examine Amazon IoT reviews - one of the biggest IoT markets. This work presents a state-of-the-art methodology to identify and categorize reviews in which users express S&P concerns. We developed an automated pipeline by fine-tuning GPT-3.5-Turbo to build two models: the Classifier-Rationalizer-Categorizer and the Thematic Mapper. By leveraging dynamic few-shot prompting and the model's large context size, our pipeline achieved over 97% precision and recall, significantly outperforming keyword-based and classical ML methods. We applied our pipeline to 91K Amazon reviews about fitness trackers, smart speakers and cameras, over multiple years. We found that on average 5% contained S&P concerns, while security camera exhibited the highest prevalence at 10%. Our method detected significantly more S&P-relevant reviews than prior works: 15x more for fitness trackers, 29% more for smart speakers, and 70% more for cameras. Our longitudinal analysis reveals that concerns like surveillance and data control have persisted for years, suggesting limited industry progress. We demonstrate that across all device types, users consistently demand more precise control over what data is collected and shared. We uncover challenges in multi-user and multi-device interactions, identifying two previously unreported themes concerning inadequate controls for account separation and data access. These findings, ranging from broad persistent trends to specific instances of customer loss, offer actionable insights for developers to improve user satisfaction and trust.