ScreenTK: Seamless Detection of Time-Killing Moments Using Continuous Mobile Screen Text and On-Device LLMs
Le Fang, Shiquan Zhang, Hong Jia, Jorge Goncalves, Vassilis Kostakos

TL;DR
ScreenTK introduces a novel on-device method that uses continuous screen text analysis and large language models to more accurately detect time-killing moments on smartphones, surpassing previous screenshot-based approaches.
Contribution
The paper presents a new framework called ScreenTK that leverages continuous screen text and on-device LLMs to detect time-killing moments more effectively than existing screenshot-based methods.
Findings
ScreenTK outperforms state-of-the-art solutions by 38%.
Up to 50% of time-killing instances are missed by screenshot methods.
Experiments with six participants and 1,034 records validate the approach.
Abstract
Smartphones have become essential to people's digital lives, providing a continuous stream of information and connectivity. However, this constant flow can lead to moments where users are simply passing time rather than engaging meaningfully. This underscores the importance of developing methods to identify these "time-killing" moments, enabling the delivery of important notifications in a way that minimizes interruptions and enhances user engagement. Recent work has utilized screenshots taken every 5 seconds to detect time-killing activities on smartphones. However, this method often misses to capture phone usage between intervals. We demonstrate that up to 50% of time-killing instances go undetected using screenshots, leading to substantial gaps in understanding user behavior. To address this limitation, we propose a method called ScreenTK that detects time-killing moments by…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsUser Authentication and Security Systems
