Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

TL;DR

This paper introduces MONDAY, a large-scale dataset of mobile OS navigation videos, and an automated framework for expanding such datasets, significantly improving cross-platform mobile agent performance.

Contribution

It presents the MONDAY dataset and an automated video collection framework, enabling scalable, annotation-free dataset expansion for mobile OS navigation tasks.

Findings

Models trained with MONDAY outperform those trained on existing datasets by 18.11% on unseen platforms.

The framework achieves 95.04% F1score in OCR-based scene detection.

UI element detection hit ratio is 99.87%.

Abstract

Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have sparked significant interest in developing GUI visual agents. We introduce MONDAY (Mobile OS Navigation Task Dataset for Agents from YouTube), a large-scale dataset of 313K annotated frames from 20K instructional videos capturing diverse real-world mobile OS navigation across multiple platforms. Models that include MONDAY in their pre-training phases demonstrate robust cross-platform generalization capabilities, consistently outperforming models trained on existing single OS datasets while achieving an average performance gain of 18.11%p on an unseen mobile OS platform. To enable continuous dataset expansion as mobile platforms evolve, we present an automated framework that leverages publicly available video content to create comprehensive task datasets without manual annotation. Our framework comprises robust OCR-based scene detection (95.04% F1score), near-perfect UI element detection (99.87% hit ratio), and novel multi-step action identification to extract reliable action sequences across diverse interface configurations. We contribute both the MONDAY dataset and our automated collection framework to facilitate future research in mobile OS navigation.