Writing in The Air: Unconstrained Text Recognition from Finger Movement Using Spatio-Temporal Convolution

TL;DR

This paper introduces a new WiTA dataset capturing finger movements for unconstrained text recognition in air writing, along with a spatio-temporal CNN model achieving real-time performance for HCI applications.

Contribution

It provides a large, multilingual WiTA dataset and proposes a novel spatio-temporal residual network for real-time air-writing recognition, advancing HCI and vision-NLP integration.

Findings

Achieved real-time decoding speeds of 435 and 697 fps for Korean and English.

Created a comprehensive WiTA dataset with over 200,000 video instances.

Demonstrated the effectiveness of spatio-temporal CNNs for unconstrained air-writing recognition.

Abstract

In this paper, we introduce a new benchmark dataset for the challenging writing in the air (WiTA) task -- an elaborate task bridging vision and NLP. WiTA implements an intuitive and natural writing method with finger movement for human-computer interaction (HCI). Our WiTA dataset will facilitate the development of data-driven WiTA systems which thus far have displayed unsatisfactory performance -- due to lack of dataset as well as traditional statistical models they have adopted. Our dataset consists of five sub-datasets in two languages (Korean and English) and amounts to 209,926 video instances from 122 participants. We capture finger movement for WiTA with RGB cameras to ensure wide accessibility and cost-efficiency. Next, we propose spatio-temporal residual network architectures inspired by 3D ResNet. These models perform unconstrained text recognition from finger movement, guarantee a real-time operation by processing 435 and 697 decoding frames-per-second for Korean and English, respectively, and will serve as an evaluation standard. Our dataset and the source codes are available at https://github.com/Uehwan/WiTA.