A General Framework for the Recognition of Online Handwritten Graphics

TL;DR

This paper introduces a versatile graph-based framework for recognizing online handwritten graphics, effectively integrating symbol and structural information, adaptable to various graphic types, and balancing recognition accuracy with computational efficiency.

Contribution

The framework models graphics as labeled graphs and formulates recognition as a graph parsing problem, enabling flexible and accurate interpretation of diverse handwritten graphics.

Findings

Achieves state-of-the-art accuracy in mathematical expression recognition

Successfully applies to flowchart recognition tasks

Offers a flexible approach adaptable to different graphic notations

Abstract

We propose a new framework for the recognition of online handwritten graphics. Three main features of the framework are its ability to treat symbol and structural level information in an integrated way, its flexibility with respect to different families of graphics, and means to control the tradeoff between recognition effectiveness and computational cost. We model a graphic as a labeled graph generated from a graph grammar. Non-terminal vertices represent subcomponents, terminal vertices represent symbols, and edges represent relations between subcomponents or symbols. We then model the recognition problem as a graph parsing problem: given an input stroke set, we search for a parse tree that represents the best interpretation of the input. Our graph parsing algorithm generates multiple interpretations (consistent with the grammar) and then we extract an optimal interpretation according to a cost function that takes into consideration the likelihood scores of symbols and structures. The parsing algorithm consists in recursively partitioning the stroke set according to structures defined in the grammar and it does not impose constraints present in some previous works (e.g. stroke ordering). By avoiding such constraints and thanks to the powerful representativeness of graphs, our approach can be adapted to the recognition of different graphic notations. We show applications to the recognition of mathematical expressions and flowcharts. Experimentation shows that our method obtains state-of-the-art accuracy in both applications.