InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write

TL;DR

InkSight introduces a novel vision-language model that converts offline handwritten notes into online digital ink, effectively bridging the gap between traditional pen-and-paper and digital note-taking with high accuracy and generalization.

Contribution

This work is the first to successfully derender handwritten text in diverse photos and sketches using a combined reading and writing prior, without requiring large paired datasets.

Findings

87% of outputs are valid tracings on HierText dataset

67% of outputs resemble human pen trajectories

Effective generalization beyond training domain

Abstract

Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.