SeeHow: Workflow Extraction from Programming Screencasts through Action-Aware Video Analytics

TL;DR

This paper introduces SeeHow, a computer vision-based tool that automatically extracts detailed programming workflows from screencasts, enabling better understanding and interaction with programming tutorials and live coding videos.

Contribution

The work presents a novel CV-based approach for extracting fine-grained code editing steps from screencasts, improving accessibility and analysis of programming workflows.

Findings

Accurately extracts code-line editing steps from screencasts

Extracted workflows are intuitive for developers to understand

Demonstrates effectiveness on 41 hours of diverse videos

Abstract

Programming screencasts (e.g., video tutorials on Youtube or live coding stream on Twitch) are important knowledge source for developers to learn programming knowledge, especially the workflow of completing a programming task. Nonetheless, the image nature of programming screencasts limits the accessibility of screencast content and the workflow embedded in it, resulting in a gap to access and interact with the content and workflow in programming screencasts. Existing non-intrusive methods are limited to extract either primitive human-computer interaction (HCI) actions or coarse-grained video fragments.In this work, we leverage Computer Vision (CV) techniques to build a programming screencast analysis tool which can automatically extract code-line editing steps (enter text, delete text, edit text and select text) from screencasts.Given a programming screencast, our approach outputs a sequence of coding steps and code snippets involved in each step, which we refer to as programming workflow. The proposed method is evaluated on 41 hours of tutorial videos and live coding screencasts with diverse programming environments.The results demonstrate our tool can extract code-line editing steps accurately and the extracted workflow steps can be intuitively understood by developers.