AutoJournaling: A Context-Aware Journaling System Leveraging MLLMs on Smartphone Screenshots

TL;DR

AutoJournaling is a novel system that automatically generates personal journals by analyzing smartphone screenshots, leveraging digital phenotyping to provide insights into psychological states with high feasibility and accuracy.

Contribution

It introduces the first framework to use seamlessly collected smartphone screenshots for automated journal generation and psychological insight extraction.

Findings

Collected screenshots every 3 seconds from students over five days.

Demonstrated system's feasibility and accuracy in journal generation.

Provides new insights into psychological states through digital phenotyping.

Abstract

Journaling offers significant benefits, including fostering self-reflection, enhancing writing skills, and aiding in mood monitoring. However, many people abandon the practice because traditional journaling is time-consuming, and detailed life events may be overlooked if not recorded promptly. Given that smartphones are the most widely used devices for entertainment, work, and socialization, they present an ideal platform for innovative approaches to journaling. Despite their ubiquity, the potential of using digital phenotyping, a method of unobtrusively collecting data from digital devices to gain insights into psychological and behavioral patterns, for automated journal generation has been largely underexplored. In this study, we propose AutoJournaling, the first-of-its-kind system that automatically generates journals by collecting and analyzing screenshots from smartphones. This system captures life events and corresponding emotions, offering a novel approach to digital phenotyping. We evaluated AutoJournaling by collecting screenshots every 3 seconds from three students over five days, demonstrating its feasibility and accuracy. AutoJournaling is the first framework to utilize seamlessly collected screenshots for journal generation, providing new insights into psychological states through digital phenotyping.