V2S: A Tool for Translating Video Recordings of Mobile App Usages into Replayable Scenarios

TL;DR

V2S is an automated tool that converts Android app usage videos into replayable scenarios by classifying user actions with neural networks, significantly reducing manual effort in analyzing software artifacts.

Contribution

This paper introduces V2S, a novel neural network-based system that automatically translates mobile app usage videos into executable scenarios, streamlining the analysis process.

Findings

Achieves approximately 89% accuracy in reproducing actions from videos

Successfully processes 175 videos with 3,534 GUI actions across diverse apps

Demonstrates robustness across different devices and usage cases

Abstract

Screen recordings are becoming increasingly important as rich software artifacts that inform mobile application development processes. However, the amount of manual effort required to extract information from these graphical artifacts can hinder resource-constrained mobile developers. This paper presents Video2Scenario (V2S), an automated tool that processes video recordings of Android app usages, utilizes neural object detection and image classification techniques to classify the depicted user actions, and translates these actions into a replayable scenario. We conducted a comprehensive evaluation to demonstrate V2S's ability to reproduce recorded scenarios across a range of devices and a diverse set of usage cases and applications. The results indicate that, based on its performance with 175 videos depicting 3,534 GUI-based actions, V2S is accurate in reproducing $\approx$89\% of actions from collected videos.