psc2code: Denoising Code Extraction from Programming Screencasts

TL;DR

This paper introduces psc2code, a comprehensive system that denoises and extracts source code from programming screencasts using CNN-based classification, image segmentation, OCR, and language models, enabling effective search and interaction tools.

Contribution

The paper presents a novel pipeline combining CNN, image segmentation, OCR, and language models specifically for extracting clean source code from screencasts, which was not previously addressed.

Findings

CNN achieves 0.95 F1-score in frame classification

Search engine precision@5 is 0.93

OCR error correction improves code accuracy

Abstract

In this paper, we propose an approach named psc2code to denoise the process of extracting source code from programming screencasts. First, psc2code leverages the Convolutional Neural Network based image classification to remove non-code and noisy-code frames. Then, psc2code performs edge detection and clustering-based image segmentation to detect sub-windows in a code frame, and based on the detected sub-windows, it identifies and crops the screen region that is most likely to be a code editor. Finally, psc2code calls the API of a professional OCR tool to extract source code from the cropped code regions and leverages the OCRed cross-frame information in the programming screencast and the statistical language model of a large corpus of source code to correct errors in the OCRed source code. We conduct an experiment on 1,142 programming screencasts from YouTube. We find that our CNN-based image classification technique can effectively remove the non-code and noisy-code frames, which achieves an F1-score of 0.95 on the valid code frames. Based on the source code denoised by psc2code, we implement two applications: 1) a programming screencast search engine; 2) an interaction-enhanced programming screencast watching tool. Based on the source code extracted from the 1,142 collected programming screencasts, our experiments show that our programming screencast search engine achieves the precision@5, 10, and 20 of 0.93, 0.81, and 0.63, respectively.