ScreenTK: Seamless Detection of Time-Killing Moments Using Continuous Mobile Screen Text and On-Device LLMs

TL;DR

ScreenTK introduces a novel on-device method that uses continuous screen text analysis and large language models to more accurately detect time-killing moments on smartphones, surpassing previous screenshot-based approaches.

Contribution

The paper presents a new framework called ScreenTK that leverages continuous screen text and on-device LLMs to detect time-killing moments more effectively than existing screenshot-based methods.

Findings

ScreenTK outperforms state-of-the-art solutions by 38%.

Up to 50% of time-killing instances are missed by screenshot methods.

Experiments with six participants and 1,034 records validate the approach.

Abstract

Smartphones have become essential to people's digital lives, providing a continuous stream of information and connectivity. However, this constant flow can lead to moments where users are simply passing time rather than engaging meaningfully. This underscores the importance of developing methods to identify these "time-killing" moments, enabling the delivery of important notifications in a way that minimizes interruptions and enhances user engagement. Recent work has utilized screenshots taken every 5 seconds to detect time-killing activities on smartphones. However, this method often misses to capture phone usage between intervals. We demonstrate that up to 50% of time-killing instances go undetected using screenshots, leading to substantial gaps in understanding user behavior. To address this limitation, we propose a method called ScreenTK that detects time-killing moments by leveraging continuous screen text monitoring and on-device large language models (LLMs). Screen text contains more comprehensive information than screenshots and allows LLMs to summarize detailed phone usage. To verify our framework, we conducted experiments with six participants, capturing 1,034 records of different time-killing moments. Initial results show that our framework outperforms state-of-the-art solutions by 38% in our case study.