What Social Media Use Do People Regret? An Analysis of 34K Smartphone Screenshots with Multimodal LLM

TL;DR

This study analyzes 34,000 smartphone screenshots using multimodal LLMs to understand social media use regret, revealing how user intention and specific activities influence regret levels.

Contribution

It introduces a novel passive data collection method combined with multimodal LLM analysis to study social media regret at a fine-grained level.

Findings

Regret varies with user intention, especially in non-intentional social media use.

Viewing algorithmic recommendations and comments increases regret.

Browsing social media when intending direct communication slightly raises regret.

Abstract

Smartphone users often regret aspects of their phone use, especially social media use. However, pinpointing specific ways in which the design of an interface contributes to regrettable use can be challenging due to the complexity of social media app features and user intentions. We conducted a one-week study with 17 Android users, using a novel method where we passively collected screenshots every five seconds, which we analyzed via a multimodal large language model to understand participants' usage activity at a fine-grained level. Triangulating this data with data from experience sampling, surveys, and interviews, we found that regret varies based on user intention, with non-intentional and social media use being especially regrettable. Regret also varies by social media activity; participants were most likely to regret viewing algorithmically recommended content and comments. Additionally, participants frequently deviated to browsing social media when their intention was direct communication, which slightly increased their regret. Our findings provide guidance to designers and policy-makers seeking to improve users' experience and autonomy.