Uncertainty-aware Evaluation of Time-Series Classification for Online Handwriting Recognition with Domain Shift

TL;DR

This paper evaluates uncertainty quantification methods for online handwriting recognition models, focusing on their ability to detect out-of-distribution data and domain shifts in spatio-temporal data.

Contribution

It provides a comprehensive assessment of Bayesian UQ techniques like SWAG and Deep Ensembles for handwriting recognition under domain shift conditions.

Findings

UQ methods effectively detect out-of-distribution samples.

SWAG and Deep Ensembles improve model reliability.

Domain shifts between right- and left-handed writers are identifiable.

Abstract

For many applications, analyzing the uncertainty of a machine learning model is indispensable. While research of uncertainty quantification (UQ) techniques is very advanced for computer vision applications, UQ methods for spatio-temporal data are less studied. In this paper, we focus on models for online handwriting recognition, one particular type of spatio-temporal data. The data is observed from a sensor-enhanced pen with the goal to classify written characters. We conduct a broad evaluation of aleatoric (data) and epistemic (model) UQ based on two prominent techniques for Bayesian inference, Stochastic Weight Averaging-Gaussian (SWAG) and Deep Ensembles. Next to a better understanding of the model, UQ techniques can detect out-of-distribution data and domain shifts when combining right-handed and left-handed writers (an underrepresented group).