Stable Electromyographic Sequence Prediction During Movement Transitions using Temporal Convolutional Networks

TL;DR

This paper demonstrates that temporal convolutional networks improve the accuracy and stability of predicting hand and wrist movements from myoelectric signals, especially during movement transitions, outperforming existing methods.

Contribution

The study introduces the use of temporal convolutional networks for myoelectric signal classification, enhancing prediction stability during movement transitions.

Findings

Temporal convolutional networks outperform state-of-the-art methods in accuracy.

They significantly improve prediction stability during movement transitions.

The approach is validated on nine human subjects.

Abstract

Transient muscle movements influence the temporal structure of myoelectric signal patterns, often leading to unstable prediction behavior from movement-pattern classification methods. We show that temporal convolutional network sequential models leverage the myoelectric signal's history to discover contextual temporal features that aid in correctly predicting movement intentions, especially during interclass transitions. We demonstrate myoelectric classification using temporal convolutional networks to effect 3 simultaneous hand and wrist degrees-of-freedom in an experiment involving nine human-subjects. Temporal convolutional networks yield significant $(p<0.001)$ performance improvements over other state-of-the-art methods in terms of both classification accuracy and stability.

Tables

$k$-NN:	$k$-nearest neighbors, $k=3$
SVM-RBF:	Support vector machine, gaussian radial basis
Forest:	Random forest
ANN:	Artificial neural network, 3 layers x 5 nodes

Equations

$${\mathbf{E}}_i=f({\mathbf{W}}_i\ast \mathbf{X}+{\mathbf{b}}_i),i=1,2,\dots ,M.$$

$${\hat{\mathbf{y}}}^{(t)}=\text{softmax}({\mathbf{UE}}^{(t)}+\mathbf{c})$$

$$\displaystyle c_{\mathbf{p}}=\sum_{i=2}^{N}\big{(}1-\delta(p_{i},p_{i-1})\big{)}$$

$$S_{p|t}=1-\frac{|c_{\mathbf{p}}-c_{\mathbf{t}}|}{N-1}$$

Full text

\pgfmathresultptCopyright (c) 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses,in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating newcollective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

I Introduction

Classification of electromyographic (EMG) signals for upper-limb prosthesis activation involves algorithmically learning to distinguish patterns in EMG signals that correlate to discrete hand, wrist, or arm motions [1]. Techniques such as support vector machines (SVM) can discriminate a large number of EMG movement-class patterns under ideal conditions [2]. EMG pattern classification algorithms typically use individual data samples, represented as extracted features from a fixed window of raw EMG, to compute the boundaries that best segment the samples into distinct movement classes.

Muscle activations during steady-state contractions are generally more stable, especially with user practice, and the EMG signal patterns tend to reliably fall into established classes [3]. However, transient-state interclass movements pose a challenge for classifiers due to the non-stationary nature of the signal patterns [4][5]. In these cases, the model’s prediction stream can exhibit segments of erratic and incorrect predictions (Fig. 1). Post-processing methods have been proposed to stabilize the prediction stream such as majority filtering and confidence-based rejection [6]. Other methods achieve stable and accurate performance by updating class boundaries adaptively [7][8] and enhancing condition-tolerance [9].

Non-sequential prediction models like SVM can behave erratically during transient-states, in part because they are denied temporal context provided by the preceding sequence of consecutive inputs. EMG windows or frames are typically predicted independently, with the windowed feature extraction technique itself serving as a compressed temporal representation of EMG. Much like photographs only capture a portion of the information about a moving subject, these models provide a rough snapshot of a dynamic system frozen in time– critical temporal context is lost in the translation.

Sequential prediction models such as long short-term memory (LSTM) recurrent networks [10] are the state-of-the-art for time-series prediction tasks like speech [11] and activity [12] recognition. At present, recurrent networks are being used for movement prediction from cortical signals [13] and EMG [14]. Herein, we present a temporal convolutional network sequential model [15] for EMG classification that is significantly more accurate and stable than prevailing sequential and non-sequential methods, especially during movement transitions.

II Methods

II-A Temporal Convolutional Networks

Temporal convolutional networks (TCN) are a class of sequential prediction models that are designed to learn hidden temporal dependencies within input sequences. The TCN model used herein consists of a single layer of convolutional filters followed by a time-distributed, fully-connected classification layer (Fig. 2). Within the convolution layer, a collection of $M$=64 convolutional filters $\mathbf{W}\in\mathbb{R}^{d\times F}$, where $d$=25 is the duration of the filter and $F$=8 is the number of input features, are convolved along the temporal dimension of input sequence $\mathbf{X}\in\mathbb{R}^{F\times T}$, where $T$ is the number of 25 ms time-steps in the sequence, to produce temporal feature maps $\mathbf{E}\in\mathbb{R}^{M\times T}$, where

[TABLE]

Rectified linear unit (ReLU) activations [16] are applied to each element which are fed to a time-distributed, fully-connected layer for classification. Softmax activation [17] is applied to produce $C$ class probabilities for current time $t$

[TABLE]

where $\mathbf{U}\in\mathbb{R}^{C\times M}$ and $\mathbf{c}\in\mathbb{R}^{C\times 1}$ are the output weight matrix and bias, respectively. During preliminary testing, a subject (excluded from results) performed the experiment described in Sec. II-C. TCN and LSTM sequential models were trained from the first 3 min of data and tested on the last 3 min to determine optimal window sizes, input sequence lengths, and TCN dimensions. Performance contours for TCN and LSTM model parameters are shown in Fig. 3.

II-B Assessing Prediction Stability

In addition to performance accuracy, we wish to quantify a sense of the stability of a model, or how inclined the model is towards erratic class-switching or misclassification during volitional class-to-class movement. Furthermore, a model may cleanly and appropriately switch between classes, but with slight delay or anticipation. The accuracy metric penalizes this generally benign behavior, often more-so than erratic behavior.

To complement performance accuracy, we define a stability metric to quantify a model’s class-switching behavior relative to ground truth behavior. Given a vector $\mathbf{p}$ representing a series of $N\in\mathbb{Z}_{>1}$ predictions over time, we can count how many times that the prediction model switches its class output with

[TABLE]

where $\delta(\cdot,\cdot)$ is the Kronecker delta function, or equivalence indicator. After computing $c_{\mathbf{p}}$ as well as $c_{\mathbf{t}}$ from ground truth label sequence $\mathbf{t}$, our prediction stability metric

[TABLE]

quantifies how varied predictions are relative to ground truth. Ideally, $S_{\mathbf{p|t}}=1$, meaning a model’s prediction output changes exactly as often as ground truth changes. Together, accuracy and our stability metric provide a fuller understanding of a model’s behavior. Our later comparison of TCN and LSTM highlights the potential for disparity between these metrics.

II-C Experimental Methods

This study was conducted in accordance with protocols approved by the Johns Hopkins Medicine Institutional Review Boards. 9 able-bodied subjects (8 male, 1 female) participated in these experiments, ages: 24.1$\pm$3.2 years. Most subjects were inexperienced with EMG classification.

II-C1 Data Acquisition, Prediction, and Visualization

Eight channels of raw EMG sampled at 200 Hz were obtained from a Myo Armband (Thalmic Labs, Ontario, Canada) placed around the circumference of forearm muscle of greatest mass. Hand grasp position data were recorded with the Cyberglove II (CyberGlove Systems LLC, San Jose, CA). Wrist position data were recorded using two 9-axis MPU-9150 inertial sensors (InvenSense, San Jose, CA) (Fig. 4A).

For non-sequential models, every 25 ms we used a 200 ms sliding window to extract time-domain (TD5) features from the raw EMG signals: mean absolute value (MAV), waveform length, variance, slope sign change, and zero crossings [1]. For sequential models TCN and LSTM, the optimal sliding window sizes were 200 ms and 175 ms, respectively. We observed that sequential models often performed best with only MAV features trained 35 epochs. In our results, we compared the TCN model (MAV only) with LSTM (MAV), as well as the following non-sequential prediction models (TD5):

A user interface was developed for Python to control the virtual Modular Prosthetic Limb (vMPL) subcomponent of Johns Hopkins University Applied Physics Laboratory’s MiniVIE system [18][19] in order provide the subject with a real-time display of their current hand/wrist orientations (Fig. 4B).

II-C2 3-DOF Simultaneous Protocol

After an initial 15 min practice session to familiarize the subject with movement classes and contraction consistency, subjects were asked to explore for 40 s their full range of motion in each of the 3 DOFs representing wrist and hand movements: rest, hand open/close, wrist flexion/extension, and radial/ulnar flexion (Fig. 4C). Outer boundary positions $\theta^{min}_{i}$, $\theta^{max}_{i}$, and comfortable interior rest position $\theta^{0}_{i}$ were determined for each DOF $i$ from this calibration period. Class thresholds along each DOF were set at 50% of the distances from $\theta^{0}_{i}$ to $\theta^{min}_{i}$ and $\theta^{max}_{i}$. In other words, for the ground truth to switch from “rest” to “hand close,” the hand must be moved more than half the distance from rest to $\theta^{max}_{1}$. Thus, for every time step, each DOF records a ternary encoding (Fig. 4D) allowing for $27$ distinct simultaneous 3-DOF movement classes. We determined when the subject was in a transient-state by calculating where joint-velocity magnitude exceeded a threshold (shown as red in Fig. 4E).

Subjects were instructed to explore 3-DOF simultaneous movements held at consistent, repeatable contraction levels in freeform fashion (any order, combination, and duration $<$5 s) for 6 min while EMG and position data were recorded. In our analyses, the first 3 min of this EMG data were used for model training and the last 3 min for model testing. Since the boundaries of each $\theta_{i}$ were subject-determined, there was no need for movement-cue presentation nor a guarantee that the subject would attempt all 27 3-DOF combinations.

II-C3 Experiment Analysis and Evaluation

All computations were performed with common Python 3.6.5 modules and the TemporalConvolutionalNetworks [20] open-source package. Statistical $p$-values were computed from one-way analyses of variance (ANOVA) comparing TCN accuracy and stability with other models. Figure error-bars represent 1 standard error.

III Results

Aggregated results from our 3-DOF simultaneous experiment, including $p$-values, are shown in Fig. 5. In general, TCN using only MAV features demonstrated significantly higher transient-state accuracy than non-sequential models using TD5 features: $p<0.001$ for $k$-NN; $p<0.01$ for Random forest and ANN; and $p<0.05$ for SVM-RBF (Fig. 5B). Furthermore, the stability of TCN was significantly higher than all other sequential and non-sequential models tested in both steady-states and transient-states (Fig. 5C). Importantly, the TCN and LSTM sequential models were similarly accurate, but LSTM was one of the least stable models.

For the most stable models (TCN, SVM, Forest, ANN), we observed that steady-state predictions were somewhat less stable on average (though consistently more accurate) than transient-states. These differences were not significant, but could indicate that some models become more unstable during pre-transition activation or post-transition recovery than during the physical transition itself. Examples of pre-transition instability can be seen in Figs. 1 and 5A.

IV Discussion and Conclusions

Instability in the prediction output stream is a well-known problem in EMG classification, particularly during transient interclass movement. Past attempts to mitigate instability such as majority filters and confidence-based rejection [6] focus primarily on post-processing the output of the classification model. In the specific case of majority filtration, the cost for improved stability is a prediction delay. Coupled with the EMG window length, this delay may be quite perceptible to the user. Confidence-based rejection is highly useful because it creates almost no time delay and can be appended to improve the stability of any model, including TCN.

To address inherent model stability, we hypothesized that sequential models, designed to utilize the temporal context of sequential input data, would significantly improve EMG classification compared with traditional non-sequential models (Fig. 5A). Notably, sequential models yielded better performance accuracy when provided with only the MAV feature for each channel, whereas all non-sequential models preferred the TD5 feature set (Fig. 5B). This indicates that the hidden temporal features learned from MAV sequences are equally or more valuable for EMG classification than non-sequential prediction of TD5 features.

TCN and LSTM perform similarly with respect to classification accuracy (Fig. 5B), but TCN provides significantly more stable output behavior compared to all models evaluated during both steady-states and transient-states (Fig. 5C). Therefore, though some loss in TCN accuracy is due to anticipation or delay in the timing of class-switching, its consistently stable behavior is very desirable for reliable control of upper-limb prostheses.

For natural prosthesis control, it is necessary to accurately predict during dynamic transient-states because our limbs are often in motion, between states, not merely switching discretely between fixed positions. The ability of TCN models to correctly predict during transient-state movements hints at very promising behavior when applied to multi-DOF regression, a research avenue we are currently exploring.

Acknowledgment

We wish to acknowledge Johns Hopkins Applied Physics Laboratory (JHU/APL) for developing and making available the Virtual Integration Environment (VIE), which was developed under the Revolutionizing Prosthetics program. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. N66001-10-C-4056. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DARPA and JHU/APL. We thank the human subjects who participated in this study and our colleagues Dr. Brock Wester, Robert Armiger, Dr. Colin Lea, Tae Soo Kim, and Dr. Austin Reiter.