# Recognizing Handwritten Source Code

## TL;DR

This paper investigates handwriting recognition for programming on touchscreen devices, introducing a dataset of handwritten Python code, analyzing recognition errors, and enhancing accuracy by integrating programming language grammar rules into the recognizer.

## Contribution

It provides a publicly available dataset of handwritten Python code and demonstrates improved recognition accuracy by incorporating programming grammar rules into the recognition process.

## Key findings

- Achieved 8.6% word error rate and 3.6% character error rate
- Recognition accuracy surpasses standard handwriting systems
- Outperforms virtual keyboard typing for source code input

## Abstract

Supporting programming on touchscreen devices requires effective text input and editing methods. Unfortunately, the virtual keyboard can be inefficient and uses valuable screen space on already small devices. Recent advances in stylus input make handwriting a potentially viable text input solution for programming on touchscreen devices. The primary barrier, however, is that handwriting recognition systems are built to take advantage of the rules of natural language, not those of a programming language. In this paper, we explore this particular problem of handwriting recognition for source code. We collect and make publicly available a dataset of handwritten Python code samples from 15 participants and we characterize the typical recognition errors for this handwritten Python source code when using a state-of-the-art handwriting recognition tool. We present an approach to improve the recognition accuracy by augmenting a handwriting recognizer with the programming language grammar rules. Our experiment on the collected dataset shows an 8.6% word error rate and a 3.6% character error rate which outperforms standard handwriting recognition systems and compares favorably to typing source code on virtual keyboards.