Handwritten Code Recognition for Pen-and-Paper CS Education
Md Sazzad Islam, Moussa Koulako Bala Doumbouya, Christopher D., Manning, Chris Piech

TL;DR
This paper presents novel OCR and language modeling techniques for accurately recognizing handwritten code, significantly reducing errors and supporting CS education through accessible handwritten programming recognition.
Contribution
It introduces a combined OCR and indentation recognition system with error correction and a multimodal language model for end-to-end handwritten code recognition, surpassing existing methods.
Findings
Error rate reduced from 30% to 5%
Minimal hallucination of logical fixes
Supports pedagogical research and accessibility
Abstract
Teaching Computer Science (CS) by having students write programs by hand on paper has key pedagogical advantages: It allows focused learning and requires careful thinking compared to the use of Integrated Development Environments (IDEs) with intelligent support tools or "just trying things out". The familiar environment of pens and paper also lessens the cognitive load of students with no prior experience with computers, for whom the mere basic usage of computers can be intimidating. Finally, this teaching approach opens learning opportunities to students with limited access to computers. However, a key obstacle is the current lack of teaching methods and support software for working with and running handwritten programs. Optical character recognition (OCR) of handwritten code is challenging: Minor OCR errors, perhaps due to varied handwriting styles, easily make code not run, and…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHandwritten Text Recognition Techniques · Teaching and Learning Programming · Open Education and E-Learning
