Neural-enhanced motion-to-EMG: refining simulated muscle activity from musculoskeletal models using a Seq2Seq approach
Tatsuya Teramae, Takamitsu Matsubara, Tomoyuki Noda, Jun Morimoto
TL;DR
This paper introduces a new framework to improve simulated muscle activity estimates by reducing errors in timing and spatial patterns.
Contribution
The novel STDR-Net uses a Seq2Seq model with attention to refine spatio-temporal distortions in muscle activity simulations.
Findings
NEM2E improved accuracy for all five muscles in a running dataset.
The framework improved estimates for two of five muscles in a walking dataset.
The method shows potential for enhancing personalized muscle activity analysis.
Abstract
Electromyography (EMG) is essential for accurate assessment of motor function in rehabilitation, sports science, and robotics. However, its various time-consuming human operations (e.g., electromagnetic noise countermeasures) limit its widespread use. Meanwhile, motion capture technology has become more accessible, leading to increasing interest in musculoskeletal simulation models such as OpenSim. Although advances have been made in individualizing the model parameters, accurately estimating muscle activity remains a significant challenge. Previous efforts to optimize the parameters in musculoskeletal model simulators have yielded limited improvements in estimation accuracy. A key source of error that is identified in this study is the spatio-temporal distortion between the estimated and actual muscle activity, which has been inadequately addressed in previous research. To address this…
Genes, proteins, chemicals, diseases, species, mutations and cell lines named across the full text — each resolved to its canonical identifier and authoritative record.
Click any figure to enlarge with its caption.
FIGURE 1
FIGURE 2
FIGURE 3
FIGURE 4
FIGURE 5
FIGURE 6
FIGURE 7| Dataset | Motion | Subject no. | Model | Muscle dim |
|---|---|---|---|---|
| Walking ( | Four different speeds | 8 | gait2392 | 92 |
| Running ( | Four different speeds | 10 | Original ( | 102 |
| Soleus | bi | Sem | tib | Rect | |
|---|---|---|---|---|---|
| One-Quarter error |
| 0.30 (p = 0.15) |
|
| 0.19 (p = 0.19) |
| 95% CI | [-0.37, −0.02] | [-0.13, 0.73] | [-0.17, 0.10] | [-0.16, 0.02] | [-0.13,0.52] |
| Effect size, | −0.92, 0.41 | 0.57, 0.12 | −0.21, 0.22 | −1.13, 0.32 | 0.51, 0.10 |
| One-Entire error |
| 0.11 (p = 0.15) |
|
| 0.19 (p = 0.26) |
| 95% CI | [-0.48, −0.18] | [-0.05, 0.28] | [-0.18, 0.04] | [-0.18, 0.04] | [-0.17,0.55] |
| Effect size, | −1.84, 0.56 | 0.57, 0.12 | −0.56, 0.19 | −0.55, 0.30 | 0.43, 0.16 |
| All-Entire error |
| 0.15 (p = 0.062) | 0.015 (p = 0.82) |
|
|
| 95% CI | [-0.60, −0.029] | [-0.0010, 0.30] | [-0.13, 0.16] | [-0.24, −0.021] | [-0.28, −0.015] |
| Effect size, | −0.92, 0.65 | 0.78, 0.18 | 0.084, 0.12 | −1.00, 0.33 | −0.93, 0.28 |
| Soleus | Bi | Semim | Tib | |
|---|---|---|---|---|
| One-Quarter error |
|
|
| 0.16 (p = 0.097) |
| 95% CI | [-0.52, −0.26] | [-0.18, −0.1] | [-0.43, −0.28] | [-0.03, 0.35] |
| Effect size, | −2.17, 0.92 | −0.21, 0.77 | −3.33, 0.80 | 0.59, 0.70 |
| One-Entire error |
|
|
|
|
| 95% CI | [-0.55, −0.40] | [-0.33, −0.09] | [-0.51, −0.36] | [-0.20, 0.14] |
| Effect size, | −4.57, 0.80 | −1.26, 0.44 | −4.05, 0.52 | −0.14, 0.38 |
| All-Entire error |
|
|
|
|
| 95% CI | [-0.68, −0.51] | [-0.49, −0.36] | [-0.56, −0.38] | [-0.25, −0.03] |
| Effect size, | −4.92, 0.87 | −4.62, 0.59 | −3.77, 0.58 | −0.89, 0.40 |
- —Moonshot Research and Development Program 10.13039/501100020963
- —New Energy and Industrial Technology Development Organization 10.13039/501100003051
- —Japan Society for the Promotion of Science 10.13039/501100001691
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMuscle activation and electromyography studies · Balance, Gait, and Falls Prevention · Context-Aware Activity Recognition Systems
1 Introduction
Electromyography (EMG) is essential for accurate assessment of motor function in rehabilitation, sports science, and robotics. However, it is not suitable for routine measurement because it requires various time-consuming human operations, such as the application of electrodes by specialists and electromagnetic noise countermeasures, for correct measurement. In addition, it is affected by sweat and changes in skin conditions, making it unsuitable for long-term measurement. In contrast, markerless motion capture systems such as THEIA three-dimensional (3D), Azure Kinect, and mediapipe allow for easy motion measurement. Mobile force plates that can measure ground reaction forces simply by putting on shoes have also been investigated (Liu et al., 2010; Adachi et al., 2011) and commercialized by Tec Gihan Co., Ltd. (M3D Force Plate). In addition, VMOCAP (Ohashi et al., 2020), which estimates motion and muscle activity using only an RGB camera, has been proposed. Owing to these technological advances, muscle activity estimation using a musculoskeletal model simulator with measurements of the motions and ground reaction forces has become a useful system for measuring muscle activity in the elderly and in athletes who wish to reduce their measurement burden.
In this context, musculoskeletal model simulators (anybody, 2025; Dhaibaworks, 2025; ARMO, 2025; SIMM, 2025) such as OpenSim (Opensim, 2025; Delp et al., 2007) have attracted increasing attention for muscle activity estimation. A key challenge in musculoskeletal simulation is the accurate modeling of individual human dynamics. Optimizing model parameters such as the bone length, muscle force characteristics, and joint stiffness is crucial for achieving more realistic analyses (Dembia et al., 2021).
Among these simulators, OpenSim is one of the most widely used open-source platforms for musculoskeletal simulation (Seth et al., 2018; Mokhtarzadeh et al., 2023; Bedo et al., 2021). OpenSim facilitates the dissemination of research within the SimTK community and is used extensively in academic and clinical research (Reinbolt et al., 2011; Mansouri and Reinbolt, 2012; Lee and Umberger, 2016; Blache et al., 2017; Nasiri et al., 2022; Willson et al., 2023; Gao et al., 2024; Zhang et al., 2024). It supports integration with skeletal models (Saul et al., 2005; Christophy et al., 2012; Arnold et al., 2010) and optimization algorithms (Seth et al., 2018; Dembia et al., 2021) as add-on components. Recent advancements, such as Myosuite (Caggiano et al., 2022) combined with Multi-Joint dynamics with Contact (MuJoCo) (Todorov et al., 2012) have further expanded the capabilities of musculoskeletal simulations.
OpenSim operates through two key layers: torque estimation and muscle activity estimation (see Supplementary Appendix SA for details on the muscle activation estimation procedure). The torque estimation layer accurately predicts joint torques by customizing musculoskeletal models based on user-specific physical characteristics and ground reaction forces (Koller et al., 2018; Saul et al., 2005; Christophy et al., 2012; Arnold et al., 2010). Conversely, the muscle activity estimation layer solves an inverse problem to derive muscle activations from joint torques. However, this inverse problem lacks a unique solution owing to the redundancy of muscle actuators in the OpenSim musculoskeletal model. OpenSim addresses this challenge by minimizing the sum of muscle activity to achieve local optimal solutions (Thelen et al., 2003); however, this often leads to significant discrepancies between the estimated and measured muscle activities.
Several sources of error in the estimation of muscle activity estimation using OpenSim have been identified. Modeling errors, particularly in muscle dynamics, are one major cause. For example, prior research (Hamner and Delp, 2013) observed a consistent time delay of approximately 75 ms between the estimated and measured muscle activities in running datasets. In addition, other studies (Nasiri et al., 2022; Gastaldi et al., 2021; Liu et al., 2008) reported shifts in the peak of muscle activity. Another critical source of error arises from the non-unique solutions to the inverse problem of determining muscle activity from joint torque owing to the redundancy of muscle actuators, which are employed to reproduce the human musculoskeletal system. OpenSim uses Computed Muscle Control (CMC) (Thelen et al., 2003) to solve this problem; however, its objective function, which minimizes activation while achieving the target torques, does not guarantee alignment with actual muscle activity patterns. To the best of our knowledge, no previous study has generically addressed the motion-to-EMG problem using a neural network, as musculoskeletal simulators do.
Based on these findings, this study hypothesizes that errors in muscle activity estimation stem from two primary factors: temporal mismatches and spatial redundancy in musculoskeletal models. As the results of previous studies have shown, the resolution of these errors is limited only by the conventional optimization of the model parameters. Therefore, as opposed to optimizing the musculoskeletal parameters, this study explores an alternative approach by introducing a compensation model to refine the OpenSim outputs.
The Neural-Enhanced Motion-to-EMG (NEM2E) framework (Figure 1) is introduced to address the above problems. It estimates realistic muscle activity from motion and ground reaction force data by incorporating the Spatio-Temporal Distortion Refinement Network (STDR-Net). Specifically, the STDR-Net refines the muscle activity outputs of OpenSim by compensating for spatio-temporal distortion. The network leverages a Sequence-to-Sequence (Seq2Seq) model (Sutskever et al., 2014), which is a recent development in natural language and image processing, enhanced with attention mechanisms (Luong et al., 2015). The framework implements Seq2Seq models with spatial and temporal attention mechanisms to investigate the spatial and temporal error contributions separately.
NEM2E framework. NEM2E refines the muscle activity estimated from motion and ground reaction force data using OpenSim, enhancing it with the STDR-Net.
The analysis was performed using the walk and running motion of elderly people and athletes, as well as extractable discrete motions such as baseball batting and golf swinging, which are commonly applied in sports and rehabilitation science. This study used two publicly available datasets (Liu et al., 2008; Hamner and Delp, 2013) that pair the muscle activity estimations of OpenSim with actual EMG data for lower-limb walking and running motions. In addition, the accuracy was tested on unknown subjects through cross-validation between subjects based on open data to enable cross-sectional measurement.
The contributions of this study are summarized as follows.
- • The NEM2E framework is developed to enhance the realism of musculoskeletal model simulators.
- • The study hypothesizes that discrepancies between simulated muscle activity and measured EMG data are owing to spatio-temporal distortions that are inherent in conventional musculoskeletal models.
- • The STDR-Net is proposed using Seq2Seq and attention mechanisms to refine the muscle activity estimations.
- • Validation is performed using public datasets, demonstrating significant accuracy improvements for all five muscles in running data and two out of five muscles in walking data.
By addressing a significant error source in musculoskeletal modeling, this study opens pathways for further innovations in refining biomechanical simulations and integrating neural networks into computational modeling.
The remainder of this paper is organized as follows: Section 2 reviews related studies, Section 3 describes the NEM2E framework, Section 4 explains the model learning and statistical analysis, Section 5 presents the validation results, Section 6 discusses the findings, and Section 7 concludes the paper.
2 Related works
One of the key roles of OpenSim is to estimate muscle activity. Thelen et al. (2003) demonstrated that integrating joint angular acceleration feedback into the muscle activity optimization routine (static optimization) of OpenSim yields muscle activity estimates with timing similar to EMG-based measurements. This method is implemented in OpenSim as CMC.
However, estimating muscle activity from joint torque remains an ill-posed problem owing to the redundancy of actuators in musculoskeletal models. Several studies have reported discrepancies between the simulated and experimental muscle activity, which have often been attributed to changes in muscle loading and the effects of assisted movement (Nasiri et al., 2022; Gastaldi et al., 2021).
Efforts to improve the accuracy of OpenSim musculoskeletal models have followed two main approaches:
2.1 Development of sophisticated models
The first approach involves creating more detailed models. For instance, studies have developed precise models of the lumbar spine, addressing areas that are not adequately represented by the base model of OpenSim (Christophy et al., 2012; Raabe and Chaudhari, 2016). Tools such as NMSBuilder developed by G. Valente et al. have also been introduced to aid researchers in developing customized OpenSim models (Valente et al., 2017). However, there are inherent limitations to the accuracy that is achievable in simulations owing to challenges in replicating the complexity of the human body. In addition, measurements such as CT scans are required to capture individual differences accurately, which may not be feasible for all users.
2.2 Optimization of model parameters
The second approach focuses on optimizing the model parameters. OpenSim moco (Dembia et al., 2021) introduces functions to calibrate parameters such as the muscle length and maximum tension using individual user data. Although this improves the personalization, discrepancies between the estimated muscle activity and measured EMG data persist. Valente et al. proposed robust optimization methods to address uncertainties in muscle models (Valente et al., 2014). However, parameter optimization alone cannot resolve fundamental modeling errors in musculoskeletal models.
2.3 Our concept
In contrast to previous studies that focused only on optimizing the parameters or models within the OpenSim framework, this study introduces the NEM2E framework. By addressing temporal and spatial modeling errors separately, the STDR-Net compensates for these errors using a Seq2Seq model with an attention mechanism. This approach has the potential to mitigate the limitations of conventional methods and improve the accuracy of muscle activity estimation.
3 Materials and methods
This section presents an overview of the methods and experimental setup.
3.1 Methods
NEM2E framework and STDR-Net.
3.1.1 NEM2E framework
The NEM2E framework refines the muscle activity estimations generated by OpenSim (Figure 1). It starts with the standard workflow of OpenSim, which includes 1) scaling (skeletal model parameter adjustment), 2) inverse kinematics (joint angle calculation), 3) residual reduction algorithm (joint torque estimation), and 4) CMC (muscle activity estimation) (see Supplementary Appendix SA for a detailed description). The resulting muscle activity data are then refined by STDR-Net to compensate for spatio-temporal distortions that are introduced by the OpenSim musculoskeletal model.
3.1.2 STDR-Net
STDR-Net addresses spatio-temporal distortions in OpenSim musculoskeletal models. Several time-series modeling techniques (e.g., linear regression, recurrent neural networks, long short-term memory (LSTM), and the Transformer) can be applied; however, this study implements STDR-Net using a Seq2Seq model with an attention mechanism (Luong et al., 2015) because this approach leverages attention layers to analyze the spatio-temporal relationships during data refinement. A schematic is shown in Figure 2. Two distinct models, (a) and (b), are constructed to analyze the temporal and spatial distortion, respectively.
STDR-Net. STDR-Net combines the Seq2Seq model and an attention mechanism that transforms time-series data ys using encoders and decoders with LSTM. There are two types of attention mechanism: (a) an attention mechanism for temporal analysis and (b) an attention mechanism for spatial analysis.
These models use the muscle activity estimated by OpenSim as the input and the measured real muscle activity as the output , and train a neural network (encoder , decoder , and attention ) that compensates for the spatio-temporal distortions between the input and output. By analyzing the attention mechanism that is embedded in the network, the information that is used to correct the distortion can be revealed, and the contribution of spatio-temporal information to refining the estimated muscle activity can be verified. See Supplementary Appendix SB for details on the Seq2Seq with attention model.
3.2 Experiment
The proposed NEM2E framework was validated using two publicly available datasets (Liu et al., 2008): published in https://simtk.org/projects/mspeedwalksims and the datasets used in the literature (Hamner and Delp, 2013) published in https://simtk.org/projects/nmbl_running. The contents of the dataset are shown in Table 1 These datasets contain the estimated muscle activation results for which individual body parameter tuning and other standard tuning has already been performed in the respected studies.
3.2.1 Model learning
The parameters of the STDR-Net were trained using a dataset, where the simulated muscle activation from OpenSim served as the input and the muscle activation obtained from the measured EMG served as the output. The squared error was used as the loss function and the optimization process was carried out using the Adam algorithm (Kingma and Ba, 2015).
For all conditions, the STDR-Net was configured with 20 units corresponding to the number of muscle actuators in OpenSim for the encoder (input layer), 700 units for each intermediate layer, and 20 units for the decoder (output layer). Training was performed with a batch size of 1 and 700 epochs. The data were standardized by thinning the input and output data to 20 samples each. For the temporal attention models, the input dimension corresponded to the single dimension of the target muscle.
3.2.2 Statistical analysis
The performance of the proposed model was compared with the baseline muscle activity estimations of OpenSim. All EMG measurements in each dataset (running data: 16 muscles, walking data: 5 muscles) were verified. The names and abbreviations of each muscle are as follows: the soleus (soleus), biceps femoris long head (bi), semimembranosus (sem), tibialis anterior (tib), rectus femoris (rect), gluteus maximus (glmax1, glmax2, and glmax3), gluteus medius (glmed1, glmed2, and glmed3), gastrocnemius medial head (gasmed), gastrocnemius lateral head (gaslat), vastus lateralis (vaslat), and vastus medialis (vasmed) muscles. These names follow the column labels used in publicly available OpenSim muscle activity datasets and may differ from standard anatomical abbreviations. All data were segmented into individual gait cycles. A leave-one-out cross-validation approach was used, with data blocked by subject for both the walking and running datasets.
The accuracy of the model was assessed using the root mean squared error (RMSE) between the model output and measured EMG data , which is calculated using the following (Equation 1):
where is the number of samples. Assuming that the error in the refined muscle activity obtained by the proposed method is and the error in the estimated muscle activity obtained from OpenSim is , the error improvement rate is calculated as
Because each subject has data for four different speeds, the average of the four RMSE values is treated as the representative value for each subject. The null hypothesis was tested using a T-test. The effect size is calculated using Cohen’s d in the following (Equation 3).
where, and denote average and standard deviation of , respectively. is hypothesized population mean.
The 95% confidence interval (CI) is calculated using the following (Equation 4).
where, represents the two-tailed t-value of in the t-distribution with n-1 degrees of freedom. n is sample number. is 0.05. Three input data patterns were defined for validation.
- • One muscle, quarter cycle (One-Quarter): Sequence data for a quarter cycle of one muscle of the target muscle.
- • One muscle, entire-cycle (One-Entire): Sequence data for an entire cycle of one muscle of the target muscle.
- • All muscles, entire cycle (All-Entire): Sequence data for an entire cycle of all muscle sets in the musculoskeletal model.
One-Entire tested the temporal distortion compensation. Because the proposed framework emphasizes the importance of utilizing the entire sequence data to address temporal distortion, the effect of using only a portion of the sequence as input was examined in the One-Quarter model. All-Entire assessed the spatial distortion compensation by incorporating all muscle activity data. Temporal attention (Figure 2a) was used for One-Entire and One-Quarter, whereas spatial attention (Figure 2b) was applied to All-Entire. Tukey’s multiple comparison tests were performed for each muscle to analyze significant differences among the three input patterns.
4 Results
4.1 Refinement results of NEM2E
Figure 3 shows representative results from the NEM2E framework. The results for the One-Entire condition in the walking dataset are shown in (a) and the results for the All-Entire condition in the running dataset are shown in (b). The blue lines indicate simulated muscle activity from OpenSim, the red lines show the refined output from the STDR-Net, and the black lines represent measured EMG data. The horizontal axis denotes the gait cycle (right heel strike to left heel strike), whereas the vertical axis represents muscle activation. In all cases, the refined muscle activation (red) closely matched the measured EMG (black), demonstrating the ability of the framework to improve upon the baseline OpenSim estimates (blue).
Refined muscle activities with our proposed method. (a) shows the walking dataset results for the One-Entire condition. (b) shows the running dataset results for the All-Entire condition. The blue, red, and black lines represent simulated, refined, and measured muscle activity. The horizontal axis represents the gait cycle (right heel strike to left heel strike), whereas the vertical axis represents muscle activity.
4.2 Statistical analysis
Tables 2 and 3 summarize the statistical results for the improvement rates in the estimation accuracy for the walking and running datasets, respectively. indicates that the null hypothesis is rejected in the estimation error improvement rate for each muscle calculated by Equation (2). In the walking data, significant improvements in accuracy were observed for the soleus, tib, and rect under the All-Entire condition. No significant improvements were observed for bi and sem under any conditions. In the running dataset, significant improvements were evident for all muscles in the All-Entire condition. Under the One-Quarter condition, only 4 out of the 16 muscles showed significant improvement, and under the One-Entire condition, 10 out of the 16 muscles demonstrated significant improvement.
Figure 4 show the improvement rates in the estimation error for the walking and running datasets. A value of 0 represents the baseline accuracy, whereas negative values indicate improvement. In the walking dataset, there were no significant differences between conditions. Conversely, in the running data, significant differences were observed between One-Quarter and All-Entire in 13 out of 16 muscles as well as between One-Entire and All-Entire in bi and glmax2.
*Comparison of estimation accuracy improvement. (a–c) show the results for the running dataset, and (d) shows the results for the walking dataset. The green, blue, and gray boxes represent the One-Entire, One-Quarter, and All-Entire conditions, respectively. The horizontal axis indicates target muscles. The vertical axis represents the error rate Er . Statistical significance is indicated by *p <0.05 , **p <0.01 , and **p <0.001 (Tukey’s multiple comparisons test).
In the running dataset, significant improvements were observed in all muscles under the All-Entire condition. However, under the One-Entire condition, no significant improvements were observed in 6 out of the 16 muscles. Furthermore, in the walking dataset, significant improvements in the accuracy of the gastrocnemius muscle were observed under all conditions. The rectus femoris muscle showed improvements only under the All-Entire condition, suggesting that the spatial attention mechanism in this condition plays a crucial role in enhancing muscle activity estimation. Furthermore, under the One-Quarter condition, significant improvements were observed in only 5 out of 21 muscles across both running and walking data, indicating the need to consider the entire sequence rather than applying window processing to the sequence data. Moreover, as the models in this study were trained only on running and walking data, shortcut learning may have occurred, in which average muscle activity patterns were learned that are dependent on the movement duration. Then, for the One-Entire model trained on the soleus walk data, which showed strong improvement results in the validation, the improvement rate of the refined results was tested with a T-test by entering 32 randomly frequency- and phase-changed data, which was the same number of test trials in the walk dataset of the original paper. The average improvement rate (- indicates improvement and + indicates degradation) was +0.48, with a p-value of 0.001, showing significant degradation. Thus, the results demonstrate that shortcut learning, which is dependent on the operating time, did not occur.
Furthermore, Figure 5 shows the loss trends for the training and validation data across each learning trial. In the running data, as shown in Figure 5a, the validation loss converged, whereas, in the walking data, as shown in Figure 5b, the loss temporarily increased without decreasing and then converged. For the running data, the validation loss demonstrates a stable relationship with the training loss, consistently decreasing alongside it. This trend suggests that overfitting is unlikely for this dataset, even with the relatively high number of epochs (700). The observed generalization across individuals further indicates that the model is successfully learning the underlying patterns for this specific sequence task without memorizing the training data.
Transition of loss over trials (a) is the value of LOSS when learning the soleus data of the run. The blue line is the loss for the training data, and the orange line is the LOSS for the validation data. (b) is the result for the soleus of walking.
Conversely, for the walk data, the validation loss exhibits an increasing trend from early epochs. However, the validation loss remains stagnant and elevated from the outset, suggesting that the intrinsic complexity or inherent noise within this dataset may hinder the model’s ability to achieve substantial learning improvements from the beginning. This observation implies that for the walk data, increasing the dataset size is considered more critical for improving model performance and generalization than simply extending the training duration or adjusting architectural parameters.
4.3 Attention
Figure 6 depicts two representative examples of temporal attention weights in the One-Entire condition. In (I), the attention mechanism primarily focuses on the central part of the input sequence to refine the initial portion of the output waveform of the model. Conversely (II) highlights attention on the latter part of the input sequence, corresponding to the refinement of the second half of the output waveform.
Temporal attention results. (I) and (II) show two typical examples of temporal attention. (a) Attention weight matrix, with the color bar indicating the weight magnitude (red for higher weights and blue for lower weights). The horizontal axis represents the input sample index and the vertical axis shows the output sample index. (b) Muscle activity, where the horizontal axis is the time and the vertical axis is the muscle activity. The blue lines represent the estimated muscle activity of OpenSim, the red lines indicate the refined muscle activity, and the black lines show the measured EMG.
The temporal attention results (Figure 6) reveal that the model allocated higher weights to data near the center of the input sequence when refining the muscle activity. This suggests that the temporal distortions in the estimations of OpenSim can be effectively learned and corrected by the attention mechanism. However, the lack of improvement in the One-Quarter condition for certain muscles, such as the biceps femoris long head, underscores the need for longer temporal sequences to capture the full distortion. The fact that the attention was focused on the peak shift in the attention weight results suggests that the model learned as intended, correcting for temporal bias rather than movement duration.
Figure 7 depicts the spatial attention results for the tibialis anterior in the All-Entire condition. Each skeleton visualizes the gait cycle motions at 0%, 50%, and 100%, with the top 10 muscles having the highest attention weights. At 0% and 100%, corresponding to the heel strike phase, similar muscle groups with high attention weights were identified. At 50%, additional trunk and hip muscles were selected, reflecting a shift in the coordination required during the swing phase.
Spatial attention results. An example of spatial attention is shown. The vertical axis of the graph indicates the muscle activity channels and the horizontal axis represents the gait cycle. The color bar denotes the weight magnitude (yellow for higher weights and blue for lower weights). The skeleton models highlight the top 10 muscles with the highest motion and attention weights at 0%, 50%, and 100% of the gait cycle.
The spatial attention revealed that the top 10 muscles with the highest attention weights for the tibialis anterior, which exhibited improved accuracy only in the All-Entire condition, varied at 0%, 50%, and 100% of the gait cycle. This variation may be explained by differences in the active muscle groups during the stance and swing phases of gait.
In addition, the top 10 muscles with the highest attention weights included muscles from the trunk, hip, and lower back on the opposite leg, in addition to the peripheral muscles of the target muscle. The peripheral muscles may have complemented the information obtained through muscle synergy during gait. Regarding the involvement of the hip, pelvis, and trunk muscles of the opposite leg, previous studies have reported that the trailing limb angle is related to ankle moments during gait (Hsiao et al., 2016). This suggests that ankle torque is not solely generated by muscle activity in the target leg, but also involves coordinated actions with the trunk and opposite leg. Owing to the significant number of muscles associated with the hip and trunk, these regions likely act as a counterweight for ankle movement. The results suggest that incorporating information from these muscles improved the accuracy of the muscle activity estimations by accounting for the broader coordination required during gait.
5 Discussion
We have proposed the NEM2E framework, which corrects the spatio-temporal distortion between OpenSim estimated and real muscle activity. Within this framework, a refined model for estimated muscle activity was trained using Seq2Seq with attention based on open data, and the improvement in estimation accuracy for new users was verified.
The temporal attention results shown in Figure 5a indicate that attention is focused on the 50% gait cycle muscle activity from OpenSim to estimate the 0%–40% gait cycle muscle activity. This result suggests that the proposed method refines the approximately 75 ms temporal shift observed in some muscles, as reported by a previous study (Hamner and Delp, 2013) using the running dataset. Because the temporal attention results are generalized across individuals, the results support the previous study’s suggestion that some muscles exhibit a common time delay.
In this study, we specifically address the persistent issue of spatio-temporal distortion in OpenSim muscle activity estimations. Previous studies (Koller et al., 2018; Saul et al., 2005; Christophy et al., 2012; Arnold et al., 2010) have focused on model tuning within the OpenSim environment to improve the accuracy of musculoskeletal simulators. These include tuning body parameters, aligning joint torque with floor reaction force data, and accounting for uncertainty in tuning parameters, such as muscle length. However, our proposed NEM2E framework introduces a novel external refinement approach. By treating OpenSim outputs as initial estimates and subsequently correcting their spatio-temporal properties with a dedicated refinement network, we provide a complementary solution that enhances the accuracy of existing musculoskeletal models without modifying OpenSim’s core parameters. In some cases, the improvement in accuracy could not be confirmed in certain walk data. One possible explanation is the increased variation in spatio-temporal distortions due to individual differences in balance and control strategies during the double support phase of gait. This variation may have reduced the generalization performance of the model owing to the limited dataset. Because the loss in the verification data in Figure 5b does not decrease, it is necessary to increase the size of the learning data and verify its effectiveness. Increasing the dataset size could address this issue and improve the effectiveness of the model.
A limitation of this work is the use of open data for walking and running as the validation dataset. Therefore, this method can primarily be applied to discrete motions, and the accuracy of the model for entirely unknown motions cannot be guaranteed. However, as generalization to unknown users is allowed, the method can be widely used in the analysis of movements related to walking and running, such as walking analysis of the elderly and athletics. The framework can also be applied to other discrete motions, such as golf swinging and baseball batting and pitching. However, verification regarding motions other than walking and running is crucial. In the future, it is necessary to build a benchmark dataset with a wide range of motions and large volume of data to validate the proposed framework and design appropriate refinement models.
The observed improvements in estimation accuracy for new users, particularly in generalization performance, highlights the importance of explicitly addressing spatio-temporal distortions. These findings directly support the hypothesis that spatio-temporal distortions contribute significantly to errors in OpenSim’s estimations. Furthermore, the effectiveness of the Seq2Seq with attention architecture in isolating and correcting these errors indicates that data-driven models offer a promising means of enhancing the fidelity of physiologically informed models. By externally addressing these distortions through a refinement network, the proposed framework provides a promising approach for enhancing the accuracy and applicability of musculoskeletal models. As a refined model in this study, the Seq2Seq with attention model was used as an example implementation to analyze spatio-temporal distortions; however, more suitable models may exist for this design. If a sufficient dataset can be constructed to train and compare models, the proposed framework could incorporate methods that more accuratey refine the spatio-temporal distortions discussed in this study. As a future direction, if a sufficient dataset can be constructed, research on direct estimation of muscle activity using neural network methods may become feasible.
The code to train the public dataset and perform CV between subjects under the All-Entier condition of this paper is available at SimTK (https://simtk.org/projects/nem2e).
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Adachi W.Tsujiuchi N.Koizumi T.Shiojima K.Tsuchiya Y.Inoue Y. (2011). Development of walking analysis system using by motion sensor with Mobile force plate. J. Syst. Des. Dyn. 6, 655–664. 10.1299/jsdd.6.655 22255222 · doi ↗ · pubmed ↗
- 2Any Body (2025). Any Body Modeling System. Aalborg, Denmark: Any Body Technology. Available online at: https://www.anybodytech.com/.
- 3ARMO (2025). Healthcare Tech Solves Corporate and Social Issues and Improves Business Productivity. Tokyo, Japan: G-Sport Inc. Available online at: https://www.gsport.co.jp/products-2/armo/.
- 4Arnold E. M.Ward S. R.Lieber R. L.Delp S. L. (2010). A model of the lower limb for analysis of human movement. Ann. Biomed. Eng. 38, 269–279. 10.1007/s 10439-009-9852-5 19957039 PMC 2903973 · doi ↗ · pubmed ↗
- 5Bedo B. L. S.Mantoan A.Catelli D. S.Cruaud W.Reggiani M.Lamontagne M. (2021). Bops: a matlab toolbox to batch musculoskeletal data processing for opensim. Comput. Methods Biomechanics Biomed. Eng. 24, 1104–1114. 10.1080/10255842.2020.1867978 33427495 · doi ↗ · pubmed ↗
- 6Blache Y.Begon M.Michaud B.Desmoulins L.Allard P.Dal Maso F. (2017). Muscle function in glenohumeral joint stability during lifting task. P Los One 12, e 0189406. 10.1371/journal.pone.0189406 29244838 PMC 5731701 · doi ↗ · pubmed ↗
- 7Caggiano V.Wang H.Durandau G.Sartori M.Kumar V. (2022). Myosuite: a contact-rich simulation suite for musculoskeletal motor control. In: Firoozi R.Mehr N.Yel E.Antonova R.Bohg J.Schwager M., editors. Proceedings of the 4th annual learning for dynamics and control conference. New York, NY: Proceedings of Machine Learning Research, Vol. 168. p. 492–507.
- 8Christophy M.Faruk Senan N. A.Lotz J. C.O’Reilly O. M. (2012). A musculoskeletal model for the lumbar spine. Biomechanics Model. Mechanobiol. 11, 19–34. 10.1007/s 10237-011-0290-6 21318374 · doi ↗ · pubmed ↗