XMTC: Explainable Early Classification of Multivariate Time Series in Reach-to-Grasp Hand Kinematics

TL;DR

This paper introduces XMTC, an explainable ensemble method for early classification of multivariate time series in reach-to-grasp hand kinematics, with visual tools for analysis and understanding of predictions.

Contribution

The paper presents XMTC, a novel ensemble approach with integrated visualization tools for early, trustworthy classification of multivariate time series in HCI tasks.

Findings

Effective early classification achieved in real-world HCI data

Visualization tools help identify challenging conditions and impactful features

Ensemble approach improves robustness across diverse datasets

Abstract

Hand kinematics can be measured in Human-Computer Interaction (HCI) with the intention to predict the user's intention in a reach-to-grasp action. Using multiple hand sensors, multivariate time series data are being captured. Given a number of possible actions on a number of objects, the goal is to classify the multivariate time series data, where the class shall be predicted as early as possible. Many machine-learning methods have been developed for such classification tasks, where different approaches produce favorable solutions on different data sets. We, therefore, employ an ensemble approach that includes and weights different approaches. To provide a trustworthy classification production, we present the XMTC tool that incorporates coordinated multiple-view visualizations to analyze the predictions. Temporal accuracy plots, confusion matrix heatmaps, temporal confidence heatmaps, and partial dependence plots allow for the identification of the best trade-off between early prediction and prediction quality, the detection and analysis of challenging classification conditions, and the investigation of the prediction evolution in an overview and detail manner. We employ XMTC to real-world HCI data in multiple scenarios and show that good classification predictions can be achieved early on with our classifier as well as which conditions are easy to distinguish, which multivariate time series measurements impose challenges, and which features have most impact.