Fingertip Detection and Tracking for Recognition of Air-Writing in Videos

TL;DR

This paper presents a novel real-time air-writing recognition system using web-cam video, featuring a new hand pose detection, fingertip tracking with curvature entropy, and velocity-based gesture termination, achieving high accuracy and efficiency.

Contribution

It introduces a new hand pose detection and fingertip tracking method for air-writing, improving robustness and real-time performance over existing approaches.

Findings

Mean precision of 73.1% in fingertip detection

Real-time processing at 18.5 fps

Character recognition accuracy of 96.11%

Abstract

Air-writing is the process of writing characters or words in free space using finger or hand movements without the aid of any hand-held device. In this work, we address the problem of mid-air finger writing using web-cam video as input. In spite of recent advances in object detection and tracking, accurate and robust detection and tracking of the fingertip remains a challenging task, primarily due to small dimension of the fingertip. Moreover, the initialization and termination of mid-air finger writing is also challenging due to the absence of any standard delimiting criterion. To solve these problems, we propose a new writing hand pose detection algorithm for initialization of air-writing using the Faster R-CNN framework for accurate hand detection followed by hand segmentation and finally counting the number of raised fingers based on geometrical properties of the hand. Further, we propose a robust fingertip detection and tracking approach using a new signature function called distance-weighted curvature entropy. Finally, a fingertip velocity-based termination criterion is used as a delimiter to mark the completion of the air-writing gesture. Experiments show the superiority of the proposed fingertip detection and tracking algorithm over state-of-the-art approaches giving a mean precision of 73.1 % while achieving real-time performance at 18.5 fps, a condition which is of vital importance to air-writing. Character recognition experiments give a mean accuracy of 96.11 % using the proposed air-writing system, a result which is comparable to that of existing handwritten character recognition systems.