Enhancing IMU-Based Online Handwriting Recognition via Contrastive Learning with Zero Inference Overhead

TL;DR

This paper introduces ECHWR, a training framework that enhances IMU-based online handwriting recognition accuracy on edge devices by using contrastive learning with an auxiliary branch, without increasing inference overhead.

Contribution

The paper proposes a novel training method employing an auxiliary branch and dual contrastive loss to improve feature representation for handwriting recognition, maintaining efficiency during inference.

Findings

Significant reduction in character error rates on benchmark dataset.

Effective handling of unseen writing styles through error-based contrastive loss.

Maintains original model efficiency during deployment.

Abstract

Online handwriting recognition using inertial measurement units opens up handwriting on paper as input for digital devices. Doing it on edge hardware improves privacy and lowers latency, but entails memory constraints. To address this, we propose Error-enhanced Contrastive Handwriting Recognition (ECHWR), a training framework designed to improve feature representation and recognition accuracy without increasing inference costs. ECHWR utilizes a temporary auxiliary branch that aligns sensor signals with semantic text embeddings during the training phase. This alignment is maintained through a dual contrastive objective: an in-batch contrastive loss for general modality alignment and a novel error-based contrastive loss that distinguishes between correct signals and synthetic hard negatives. The auxiliary branch is discarded after training, which allows the deployed model to keep its original, efficient architecture. Evaluations on the OnHW-Words500 dataset show that ECHWR significantly outperforms state-of-the-art baselines, reducing character error rates by up to 7.4% on the writer-independent split and 10.4% on the writer-dependent split. Finally, although our ablation studies indicate that solving specific challenges require specific architectural and objective configurations, error-based contrastive loss shows its effectiveness for handling unseen writing styles.