Image-based Freeform Handwriting Authentication with Energy-oriented Self-Supervised Learning

TL;DR

This paper introduces SherlockNet, a self-supervised learning framework that enhances freeform handwriting authentication by addressing noise, high-dimensional features, and data scarcity, demonstrating robustness and efficiency in various challenging scenarios.

Contribution

The paper proposes SherlockNet, a novel energy-oriented contrastive self-supervised learning method specifically designed for robust handwriting authentication with damaged or forged data.

Findings

Outperforms existing methods on six benchmark datasets.

Effectively handles noise, damage, and forgery in handwriting data.

Requires minimal labeled data for fine-tuning.

Abstract

Freeform handwriting authentication verifies a person's identity from their writing style and habits in messy handwriting data. This technique has gained widespread attention in recent years as a valuable tool for various fields, e.g., fraud prevention and cultural heritage protection. However, it still remains a challenging task in reality due to three reasons: (i) severe damage, (ii) complex high-dimensional features, and (iii) lack of supervision. To address these issues, we propose SherlockNet, an energy-oriented two-branch contrastive self-supervised learning framework for robust and fast freeform handwriting authentication. It consists of four stages: (i) pre-processing: converting manuscripts into energy distributions using a novel plug-and-play energy-oriented operator to eliminate the influence of noise; (ii) generalized pre-training: learning general representation through two-branch momentum-based adaptive contrastive learning with the energy distributions, which handles the high-dimensional features and spatial dependencies of handwriting; (iii) personalized fine-tuning: calibrating the learned knowledge using a small amount of labeled data from downstream tasks; and (iv) practical application: identifying individual handwriting from scrambled, missing, or forged data efficiently and conveniently. Considering the practicality, we construct EN-HA, a novel dataset that simulates data forgery and severe damage in real applications. Finally, we conduct extensive experiments on six benchmark datasets including our EN-HA, and the results prove the robustness and efficiency of SherlockNet.