Development of artificial intelligence and multi-sensor-based dexterity assessment system: performance evaluation

TL;DR

This paper presents an automated system using sensors and AI to accurately assess hand dexterity, reducing reliance on expert supervision.

Contribution

A novel system combining multi-sensor data and AI for precise, automated dexterity evaluation is developed and tested.

Findings

The system successfully classified hand movements and analyzed muscle activity using EMG and IMU sensors.

Performance evaluation with 20 participants showed accurate and automatic dexterity assessment.

The system provides objective results, reducing subjectivity and delays in traditional assessments.

Abstract

Manual dexterity tests are essential for diagnosing diseases and evaluating professional skills that require fine motor control. Traditional assessments often depend on expert supervision, leading to delays, subjectivity, and inaccuracies. This study introduces an automated dexterity assessment system that integrates multiple sensors and artificial intelligence algorithms for high-precision evaluation. The system classifies hand movements, analyzes muscle contraction levels, and determines hold-release durations using electromyography (EMG), inertial measurement units (IMU), and image processing techniques. An expert system interprets the multimodal sensor data and presents the results to clinicians. A performance evaluation with 20 participants demonstrated the system’s capability to assess hand dexterity automatically and accurately.

Figures

General block diagram of the system

Comparison of test setup and classifier resultsTable 1Comparison of disc hold timesSetTotal time(s)Hold time(classifier)(s)Hold time(test setup)(s)Difference (%)17431.632.83.826127.627.21.436425.826.31.945723.825.57.154522.121.52.764419.519.40.574621.1224.384017.618.44.59391919.84.2104325.126.86.8115432.932.22.1124322.621.26.2135528.228.51.1144621.922.84.1154925.124.42.8164820.720.22.4174221.222.66.6185023.121.76.1194822.621.26.2204821.722.43.2215522.323.76.3227125.324.43.6235220.921.21.4244621.620.93.2254824.725.22.0

Images from the test of participants

General block diagram of the system

Marking hole centers on the test board

MYO armband and channel numbers

EMG signal processing stages

EMG signals during rest, maximum contraction, and hold-releaseFig. 6LLGMN network structure

Trials for the disc turning phaseFig. 10Motion classification performance test setup

Axes on the MYO armband

Trials for the disc placing phase

Trials for the disc turning phaseFig. 10Motion classification performance test setup

Full text

Introduction

The hand is one of the most used parts of the upper limb in daily life activities. Manual dexterity, defined as the ability to perform precise and coordinated movements with the hands and fingers, is crucial for performing various daily and occupational tasks. Loss of dexterity can result from conditions such as nerve injury, amputation, arthritis, fractures, carpal tunnel syndrome, muscular dystrophies, and neurological disorders [1]. The physical dimensions of the hand and fingers also affect dexterity [2, 3]. In addition, studies have shown that dexterity declines with aging [4–7]. Evaluation of manual dexterity is important in determining the functionalities and limitations in daily life activities. In terms of physiotherapy, it is necessary to determine the loss of function in the hand to determine and apply the appropriate treatment. Several standardized dexterity tests have been developed for these purposes, including the Jebsen–Taylor Hand Function Test [8], the Purdue Pegboard Test [9], the Grooved Pegboard Test, the Crawford Small Parts Dexterity Test [10], and the Minnesota Manual Dexterity Test (MMDT). These tests are widely used not only for medical assessment but also for personnel selection in industries requiring fine motor skills.

However, conventional dexterity tests have several limitations. First, they require supervision by an expert, which can lead to scheduling delays and inefficiencies in healthcare settings. Second, these tests rely primarily on total task completion time as the evaluation criterion, offering limited insights into the subcomponents of manual dexterity, such as cognitive processing time, grasping ability, and manipulation skills. Third, human supervision introduces potential measurement errors and subjectivity into the assessment process. Furthermore, individuals being tested may experience anxiety in the presence of an evaluator, which can negatively affect their performance.

Limitations of existing methods and research gaps

In recent years, researchers have attempted to develop electronic and sensor-based systems to enhance dexterity assessment. Several approaches have been explored:

Electronic pegboard tests

Acharya et al. [11] developed an electronic Nine-Hole Peg Test that detects peg insertion using light-dependent resistors (LDRs) and LEDs. However, this system only records completion time and does not analyze movement patterns or hold-release behavior. Al-Naami et al. [12] designed an electronic Grooved Pegboard Test with switches in the peg holes to measure placement time. This setup lacks motion profiling and fails to assess complex hand–eye coordination skills due to its simplified circular hole design. Okahashi et al. [13] introduced a system that tracks peg placement using magnetic switches but does not capture detailed movement kinematics or muscle activity.

Wearable sensor-based systems

Chuang et al. [14] and Rashid and Osman [15] developed dexterity assessment systems using sensor-integrated gloves. While these systems provide additional motion data, the use of gloves restricts hand movement, reducing natural dexterity. Hsiao et al. [16] developed a glove with an IMU sensor and force-sensitive resistors for dexterity assessment. Although flexible sensors are useful for determining joint range of motion, they have the disadvantages of wearing out in a short time, requiring calibration and distorting the nature of the movement. The biggest disadvantage of glove-based solutions is that gloves cause movement restriction. Riddle et al. [17] evaluated the finger forces of individuals during daily activities with pressure sensors placed on the fingers. They also investigated the effect of these sensors on the completion time of the tasks. It was found that subjects completed tasks with large objects 13% slower, medium-sized objects 31% slower, and small objects 41% slower when sensors were attached to the fingers. Woods [18] investigated the effect of wearing gloves on skill performance while undergoing MMDT. It was observed that gloves increased the test time by 35%. These results show that materials such as sensors, apparatus, cables, and gloves worn on the hand have a negative effect on dexterity.

Recent deep learning-based methods for multivariate time series classification, such as densely knowledge-aware network (DKN) [19], deep transformer capsule mutual distillation (DTCM) [20], CapMatch: semi-supervised contrastive transformer capsule [21], and deep contrastive representation learning with self-distillation [22], have shown significant performance in activity recognition and classification tasks. However, these methods primarily focus on learning global representations from time series data rather than providing explicit inference about dexterity subcomponents. While these approaches leverage feature-based knowledge distillation and self-supervised learning to improve classification accuracy, they do not incorporate multimodal sensor fusion in a way that supports detailed movement profiling for dexterity assessment. In contrast, our proposed system integrates electromyography (EMG), inertial measurement unit (IMU), and computer vision-based motion tracking to extract granular insights into hand dexterity beyond classification tasks.

Improvements and contributions of the proposed system

To address these limitations, this study presents a multi-sensor-based dexterity assessment system that integrates electromyography (EMG), inertial measurement unit (IMU), and computer vision-based motion tracking. The key contributions of this work are:

• Automated multi-sensor integration: Unlike previous works that rely on a single measurement approach, our system integrates EMG, IMU, and computer vision to capture muscle activity, movement kinematics, and dexterity subcomponent durations.
• Detailed movement and cognitive analysis: The system not only determines total test duration but also determines subtask durations (grasping, manipulation, and release) to help identify which aspect of dexterity is impaired.
• Expert rule-based system for decision support: The extracted sensor data is evaluated through an expert system that interprets dexterity impairments and provides clinically relevant insights, reducing the need for manual evaluations.
• Validation with healthy participants: The system’s performance was tested with 20 participants, demonstrating high accuracy in motion classification (96.2%) and dexterity stage recognition, ensuring reliable clinical applicability.

By integrating these advances, the proposed system provides a comprehensive, objective, and automated dexterity assessment method, addressing the limitations of traditional and existing electronic systems. The developed framework enables more accurate diagnoses, personalized rehabilitation plans, and enhanced efficiency in clinical and occupational assessments.

Material and method

The general block diagram of the system is given in Fig. 1. The system structure, the general concept of which has been revealed [23], was realized. The study was approved by the Istanbul University-Cerrahpaşa Clinical Research Ethics Committee (E-83045809–604.01.01–374132). During the MMDT, muscle contraction levels and limb velocities and accelerations are obtained through the MYO armband placed on the forearm. These are processed in MATLAB on a computer (Intel(R) Core(TM) i9-12950HX CPU @ 2.30 GHz 64 GB RAM) to obtain the times of the hold-release movement and the angular velocity and linear acceleration of the limb. The user’s placement time of the discs is determined by image processing. This process was done with Python software running on Raspberry Pi 4. In addition, the performance of the users is evaluated with the rule base, which is created with the information received from the experts, and inferences regarding the physical and cognitive status of the participant are made. In experiments, the time taken for all processes is about 140–290 ms. The details of these measurement processes, the system integration, and the detailed working principle are explained in this section.Fig. 1. General block diagram of the system

Determination of disc placing and turning times

In the study, the MMDT was used as a dexterity test setup. This test method is widely used to measure the performance of healthy hands and to monitor the patient’s progress in rehabilitation [24, 25]. The test includes 60 discs and a test board on which these discs will be placed. One surface of the discs is black, and the other surface is red. In the test, the discs must be placed in a certain order and sequence. There are two phases in the test: placing and turning. The participant is informed that time is important.

During the MMDT, image processing was used to determine the placement of the discs on the test table and the placement and rotation times of each disc. The images obtained with the Logitech C505 camera were processed in real time using the OpenCV library in Python. Images have a resolution of 640 × 480 pixels and are recorded in RGB color space. The locations of the holes on the test board where the discs were placed were marked with circles (Fig. 2). In this way, pixels with certain coordinates are scanned instead of all pixels in the camera’s field of view. Inferences were made based on the color of the pixels in the center of these holes. The table floor is white, and the disc surfaces are black and red. When the test starts, cv2.VideoCapture() is used to capture video images. cv2.COLOR_BGR2HSV is used to convert BGR to HSV. If the color of the target pixels is white, it means that there is no disc in the target yet, if the target pixel is red during the placing phase, it means that the disc is inserted correctly, and if the pixel value is in the black color range during the placing phase, it means that the disc is inserted upside down. In the turning phase, if the target pixel is red, the disc has not yet been turned; if the target pixel is black, the disc has been turned correctly; if the target pixel is white, the disc has been removed incorrectly. The color detection process was mathematically formulated using the HSV color space. Given an RGB image, the transformation to HSV is represented as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H=\left\{\begin{array}{ll}60^0\frac{G-B}{\max\;\left(R,G,B\right)-\min\;(R,G,B)}&if\;\max\;\left(R,G,B\right)=R\\60^0(2+\frac{G-B}{\max\;\left(R,G,B\right)-\min\;\left(R,G,B\right)})&if\;\max\;\left(R,G,B\right)=G\\60^0(4+\frac{G-B}{\max\;\left(R,G,B\right)-\min\;\left(R,G,B\right)})&if\;\max\;\left(R,G,B\right)=B\end{array}\right.$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H$$\end{document} is the hue angle in degrees. The thresholding for detecting red and black discs was determined by defining pixel intensity ranges in the HSV color space:Fig. 2. Marking hole centers on the test board

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{array}{l}T_{white}=\left\{H\in\left[0,180\right],S\in\left[0,30\right],V\in\lbrack220,255\rbrack\right.\\T_{red}=\left\{H\in\left[0,10\right]\cup\left[160,180\right],S\in\left[100,255\right],V\in\lbrack30,255\rbrack\right.\\T_{black}=\left\{H\in\left[0,180\right],S\in\left[0,255\right],V\in\lbrack0,30\rbrack\right.\end{array}$$\end{document}

Analysis of EMG signals and motion classification

In the system, the holding and releasing times of the discs were recorded, and it was determined how long the participant held the discs during the test. For this purpose, EMG signals from the forearm were evaluated and classified. The amplitude range of an EMG signal is 0–10 mV before amplification. The EMG signal has the most frequency power between 20 and 250 Hz. The artificial neural network-based LLGMN (log-linearized Gaussian mixture model) algorithm developed by Tsuji et al. [26] was used for motion classification. EMG signals were obtained via the Bluetooth communication protocol with the MYO armband (Fig. 3).Fig. 3MYO armband and channel numbers

With the 8-channel electrodes on the MYO armband, which can be worn on arms of different sizes with its easy-to-wear and elastic structure, muscle contraction signals are received and pre-processed. These processes are amplification, rectification, and filtering, respectively. Since the amplitude of raw EMG signals is very low, 20 dB amplification is performed. Then the negative alternans are converted to positive alternans by rectification. Finally, a 2nd-order low-pass Butterworth filter with a corner frequency of 3 Hz was used to remove noise from the signal.

The sampling frequency of the MYO Armband device is 200 Hz. Sampled signals are defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{EMG}}_{i}\left(t\right) (i=8)$$\end{document} . \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{EMG}}_{\text{i}}\left(t\right)$$\end{document} signal is normalized so that the sum of the signals from the 8 channel electrodes is 1. The normalized EMG signal ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${{EMG}^\prime}_i(t$$\end{document} )) is defined in Eq. 3.

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{EMG}}_{i}^{\prime}\left(t\right)=\frac{{\text{EMG}}_{i}\left(t\right)-{\text{EMG}}_{i}^{\text{rest}}}{\sum_{i=1}^{L}({\text{EMG}}_{i}\left(t\right)-{\text{EMG}}_{i}^{\text{res}t})} \left(L=8\right)$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{EMG}}_{i}^{\text{rest}}$$\end{document} is the mean of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{EMG}}_{i}\left(t\right)$$\end{document} in the uncontracted, released state of the arm. A diagram of EMG signal processing is given in Fig. 4.Fig. 4EMG signal processing stages

In order to perform motion classification, the muscular contraction level (MCL) needs to be calculated using the processed EMG signals. The MCL is calculated using Eq. 4.

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text{MCL}\left(t\right)=\frac{1}{N}\sum\nolimits_{n=1}^{N}\frac{{\text{EMG}}_{n}\left(t\right)-{\text{EMG}}_{n}^{\text{rest}}}{{\text{EMG}}_{n}^{\text{max}}-{\text{EMG}}_{n}^{\text{rest}}}$$\end{document}

Here \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{EMG}}_{n}^{\text{rest}}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\text{EMG}}_{n}^{\text{max}}$$\end{document} refer to the contraction levels of the muscle at rest and maximum contraction, respectively. The classification algorithm (LLGMN) formulated as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P\left(y|x\right)=\frac{\sum_{j=1}^{M}{\pi }_{j}{g}_{j}x}{\sum_{i=1}^{M}\sum_{j=1}^{M}{\pi }_{j}{g}_{j}x}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P\left(y|x\right)$$\end{document} is the posterior probability of class \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$y$$\end{document} given input \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x, {g}_{j}(x)$$\end{document} , is the Gaussian mixture function, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\pi }_{j}$$\end{document} is the weight associated with each mixture component.

The muscle contraction at rest was measured and recorded in a relaxed state, while the maximum muscle contraction was measured and recorded by squeezing the disc vigorously with the fingers. After signal preprocessing, normalization, and MCL calculation, data recording for each movement starts. The holding and releasing movements are performed between the second and sixth seconds of the 8-s recording period, and the EMG signals are recorded. An example signal is given in Fig.5. From these recorded signals, 20 samples are selected and fed to the LLGMN model. The network is trained with this data and weights are determined. Classification is performed using these weights. With the motion information obtained at the output of the network, it is determined which of the holding and releasing movements are performed. The LLGMN network structure is given in Fig. 6.Fig. 5EMG signals during rest, maximum contraction, and hold-releaseFig. 6LLGMN network structure

Analysis of arm movements

In the developed system, in addition to movement classification, arm movements were also analyzed, and the movement speed and acceleration of the arm during the test were used in the evaluation phase. For example, if the arm is motionless or moving at low speeds while the participant does not have a disc in his/her hand, it can be concluded that he/she is free thinking. Arm speed and acceleration measurements were made with a 9-axis IMU on the MYO armband. The IMU consists of a 3-axis gyroscope (range: ± 2000°/s, nonlinearity: 0.2%, Total RMS noise: 0.06°/s), 3-axis magnetometer (range: ± 1200 µT), and three-axis accelerometer (range: ± 16 g, nonlinearity: 0.5%, total RMS noise: 4 mg). Angular velocities are given as roll, pitch, and yaw, which express rotation around the X, Y, and Z axes, respectively. Figure 7 shows the position of the MYO Armband device on the arm and the axis directions.Fig. 7. Axes on the MYO armband

The acceleration is received at the sensor output in g and converted to m/s^2^. Acceleration values are recorded relative to a fixed set of axes. The gravitational acceleration of 9.81 m/s^2^ was subtracted from the acceleration measured in the Z direction. The average and maximum-minimum values of the angular velocity and linear acceleration data obtained from the IMU are presented to the experts via the interface.

Creation of expert system rule base

In the system, disc hold and release times obtained as a result of motion classification with EMG, disc placing and turning times and total test time detected by the camera, angular velocity and acceleration values obtained from the IMU are evaluated in a rule base created with the information received from experts, and an inference is presented through the interface. The rule base was created with information from a five-person expert team consisting of two physical medicine and rehabilitation physicians and three physiotherapists. The expert system was defined using a set of inference rules. Each rule is represented as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_i:IF\left(x_1\epsilon X_1\right)^\wedge\left(x_2\in X_2\right)^\wedge\dots^\wedge\left(x_n\in X_n\right)\;THEN(y\in Y)\;$$\end{document}

Sample rules are given below.

Rule < IF tt_p < max_t_p AND tt_y > min_t_p AND*** tt_t***** < max_t_t AND*** tt_t***** > min_t_t THEN Text = ′Normal′ >

For participants whose total test time in the two phases is between normal values, based on the experts’ recommendation, it will be sufficient to send a notification that the results are normal. Here, tt_p and tt_t represent the total test times in the placing and turning phases, max_t_p, min_t_p, max_t_t, and min_t_t represent the maximum and minimum normal limits in the placing and turning phases. These values are 62 s for the placing phase and 67 s for the turning phase. Normal limit values were determined according to the durations in the literature and the data obtained from 20 participants in this study. They can be changed when desired.

When the time in the placing and turning phases is above normal, the subcomponents of the test are checked by branching to the relevant functions and executing the rules there. For example, when the following rule is executed, the subcomponents of the placement phase are checked.

Rule < IF tt_p < max_t_p AND*** tt_p***** > min_t_y AND*** tt_t***** > max_t_t AND*** tt_t***** > **min_t_t >

Rule < IF rot_r_t < nrot_r_t THEN Text = ′Insufficient rotation in turning. Possible restriction of pro-sup.′ >

This sub-rule was activated when the participant completed the turning phase above normal time. As a subcomponent, the angular velocities of the arm are controlled. rot_r_t indicates the angular velocity in the roll direction during the turning phase, and nrot_r_t indicates the minimum level of the normal range of values for this velocity. If the angular velocity around the roll axis is below normal, it is indicated that the person has difficulty in the disc turning and that attention should be paid to the possibility of a limitation in the forearm pronation-supination movement.

Rule < IF acc_x_p < nacc_x_p OR*** acc_y_p***** < nacc_y_p OR*** acc_z_p***** < nacc_z_p THEN Text = ′Inadequate movement in placing. Possible restriction of the elbow and/or shoulder joint.′ >

This sub-rule was activated when the participant completed the placing phase above the normal time. acc_x_p, acc_y_p, and acc_z_p indicate the average acceleration in the x, y, and z directions, respectively. If any of these accelerations are below normal, it is indicated that the person had difficulty in reaching and that attention should be paid to the possibility of a limitation in the elbow and/or shoulder joint.

Rule < IF th_p > n_th_p THEN Text = ′High inactive waiting time.' >

This sub-rule is activated when the participant remains stationary or moves at a low level during the placement. The angular velocity and accelerations from the IMU sensor are both recorded as thinking time (th_p or th_t). If this value is normally above the maximum level of the specified range, it should be understood that the person is stuck in the test for some reason. The possibilities of not understanding the test, having a stimulus in the environment that disturbs the participant, or having a cognitive problem should be evaluated.

Rule < IF d16_t > n_d_t THEN Text = ′Delay in turning on disc 16.′ >

The sub-rules in the system check the placing and turning times of each disc. This sub-rule indicates that the participant takes longer than usual to turn the sixth disc. With this subset of rules, it can be displayed to the experts which discs are placed or turned in a longer time than normal.

In this section, example rules for different types of inferences are given and explained. More than one rule can be active for a participant at the same time. In this case, the inferences are listed on the user interface. For example, a participant may have both longer than normal disc turning times and longer than normal thinking times. The rule base of the system is open to improvement, and new rules can be added.

Results and discussion

Testing disc placing and turning times determination performance

In the study, the placing and turning times of each disc were calculated by image processing. In this process, it is important to determine the disc orientation (whether the upward-facing face is red or black). For this purpose, 100 times of placing and 100 times of turning were tested under different lighting conditions. Figure 8 shows three of the combinations in the trials. The display shows a counter for the number of discs inserted and the number of empty holes. In Fig. 8a, 15 discs are placed in random locations. The counter again shows the correct values. In Fig. 8b, one disc was placed upside down with the black side up to determine whether the system made a mistake. The system again gave the correct result. In Fig. 8c, a shadow was created on the test board and the response of the system was analyzed. In this case, the system undercounted one disc. Since the color value in the center of the bottom right disc was closer to black, this disc could not be detected. When the shadow was removed, the correct result was obtained.Fig. 8. Trials for the disc placing phase

Figure 9 shows three of the combinations during the disc rotation phase. In Fig. 9a, 10 discs were randomly flipped. The counter again shows the correct values. In Fig. 9b, a flash was shone on the test board to simulate the ambient light change. As can be seen, flashes occurred on the side of the board facing the table edge. In this case, the system could not detect the black side of one of the discs and counted one disc as missing. When the flash was turned off, the correct result was obtained. In Fig. 9c, the ambient light level was reduced. In this case, no error occurred in the system and the correct number of discs was detected.Fig. 9. Trials for the disc turning phaseFig. 10Motion classification performance test setup

In the 100 trials in the placing test, 89 correct and 11 incorrect results were obtained. In these trials, when the ambient light was normal, the correct results increased to 97 and 3 results remained incorrect. When the ambient light was changed in some of the 100 trials in the turning phase, 86 correct and 14 incorrect responses were obtained. In particular, when a strong light source was directed at the test board, reflection occurred on the black surface, resulting in incorrect results. On the other hand, reducing the ambient light did not have a negative effect. When the light was normal in these trials, the number of correct answers increased to 95. Five results remained incorrect. When the results are evaluated, a total of 87.5% accuracy was obtained in 200 trials in which the ambient light was manipulated. When the ambient light was left in its natural state, 96% accuracy was achieved in 200 trials.

Testing the performance of hold-release motion classification

In order to test the motion classification performance of the system, the setup shown in Fig. 10 was prepared. Conductive aluminum foil was wrapped around the disc. Aluminum foil was also wrapped around the user’s thumb and index finger and the two ends of a wire were connected. When the gripping motion is performed, the wire ends are short-circuited through the conductive aluminum foil on the disc and the circuit is completed. Meanwhile, “1” information is transmitted to the computer via the digital input of the V-Daq real-time data acquisition card. In the release movement, the circuit is open and “0” information is transmitted. At the same time, EMG data is also recorded via the MYO armband. Motion classification is applied to the recorded EMG signals. The classification result is compared with the hold-release states obtained from the test setup. As an example, these 2 outputs in 8 hold and release movements performed for 30 s are compared in Fig. 11. At the same time, the muscle contraction levels seen in EMG channels 4 and 5, which are dominant in the hold and release motion, are also given. With the aforementioned procedure, 20 hold-and-release movements were performed in 25 sets, and a total of 500 hold-and-release movements were recorded. Each hold-and-release gesture was performed by the users at random durations. The results are given in Table 1.Fig. 11. Comparison of test setup and classifier resultsTable 1Comparison of disc hold timesSetTotal time(s)Hold time(classifier)(s)Hold time(test setup)(s)Difference (%)17431.632.83.826127.627.21.436425.826.31.945723.825.57.154522.121.52.764419.519.40.574621.1224.384017.618.44.59391919.84.2104325.126.86.8115432.932.22.1124322.621.26.2135528.228.51.1144621.922.84.1154925.124.42.8164820.720.22.4174221.222.66.6185023.121.76.1194822.621.26.2204821.722.43.2215522.323.76.3227125.324.43.6235220.921.21.4244621.620.93.2254824.725.22.0

The total time column in Table 1 shows the time elapsed in each set (sum of hold-release), the hold time, HT (classifier), column shows the total hold time detected by the classification algorithm, and the hold time, HT (test rig), column shows the total hold time recorded by the test setup shown in Fig. 10. When the results are analyzed, it is seen that the classification algorithm reaches a minimum accuracy rate of 92.9% and a maximum accuracy rate of 99.5% when each set is analyzed separately. When these values are averaged, an accuracy of 96.2% is obtained.

The results obtained with the LLGMN algorithm are compared with the results obtained with feed forward neural networks (FNN) and support vector machine (SVM) methods. In [27], it was observed that mean absolute value (MAV) and integrated EMG (IEMG) features gave the highest accuracy value in non-real-time classification using SVM. Based on these results, MAV, IEMG, and root mean square (RMS) values were calculated from the recorded EMG data and the dataset was divided into training (15 sets) and test (10 sets). Experiments with FNN and SVM algorithms yielded the results given in Table 2. The eightfold cross-validation method was used to calculate the accuracies. Table 2. Classification results for FNN and SVM algorithmsFNNSVMRMSMAVIEMGRMSMAVIEMGAccuracy (%)81.9 ± 2.588.6 ± 1.990.1 ± 1.788.7 ± 2.196.6 ± 1.596.3 ± 1.2

When the results are analyzed, it is seen that the SVM classifier has higher accuracy than the FNN. When compared with LLGMN, it is seen that both methods give similar results. High accuracy values could be obtained because the hold-and-release gesture by its nature produces more muscle contraction than other fine sensitive finger movements.

Testing system performance with healthy participants

In order to test the measurement performance of the developed system, trials were conducted with 20 healthy participants (10 women and 10 men). Fifteen of the participants were from the white-collar occupational group (engineers, academicians, and physicians), three participants were blue-collar (technicians and servants), and 2 participants were students. The inclusion criteria of the participants were 18–60 years of age, no history of upper limb nerve amputation, upper limb bone fracture, long-term (6 months or more) immobilization of the upper limbs, no muscle disease, and predominantly right-handed. In the test, 4 repetitions were made in each phase, one of which was a trial, and their average was taken in the calculation. The images taken during the test of healthy participants are shown in Fig. 12.Fig. 12. Images from the test of participants

Analysis of disc placing and turning times

During the test, the placing and turning of the discs are monitored in real time by the camera, and the times are recorded for each phase. A second camera is used to record the tests. When the test is completed, the placing and turning times of each disc are calculated by timing the camera images with a stopwatch manually. The time error ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E}_{t}$$\end{document} ) between system measurements and manual stopwatch measurements is calculated by Eq. 7.

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E}_{T}=\frac{\left|{T}_{s}-{T}_{m}\right|}{{T}_{m}}100$$\end{document}

Here, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${T}_{s}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${T}_{m}$$\end{document} are system-measured time and manually measured time, respectively.

Table 3 provides a detailed comparison of the average placing and turning times for each of the 60 discs, calculated from data obtained across 20 participants. For each disc, average \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${T}_{s}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${T}_{m}$$\end{document} are presented. The time errors in the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${E}_{t}$$\end{document} column in the table are the average of the disc placing and turning errors for each participant. The average placing times ranged from 0.79 to 1.29 s, while average turning times ranged from 0.89 to 1.46 s. Across most discs, the system and manual measurements showed strong agreement, with average time errors generally remaining below 10%. Notable deviations above this threshold, such as in disc 2 and disc 43 during the turning phase, may be attributed to transient visual obstructions or variations in hand motion that temporarily limited the camera’s field of view. Overall, the high temporal consistency between the system and manual measurements supports the accuracy and reliability of the developed system in tracking fine motor task timings, thereby demonstrating its potential utility for objective and automated dexterity assessment. Table 3. Comparison of average disc placing, turning times, and average time error of 20 participantsPlacingTurningPlacingTurningDisc \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{s}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{m}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{E} }_{T}$$\end{document} (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{s}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{m}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{E} }_{T}$$\end{document} *(%)*Disc \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{s}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{m}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{E} }_{T}$$\end{document} (%) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{s}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{T} }_{m}$$\end{document} (s.) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\overline{E} }_{T}$$\end{document} *(%)*11.151.1311.021.441.385.30311.121.124.081.171.215.5120.990.975.021.211.0814.04321.131.144.971.501.495.8031.071.093.830.960.938.34330.940.937.651.361.365.7641.241.255.180.900.904.14340.910.876.151.141.127.2151.021.004.781.021.007.64351.021.045.861.031.046.2160.930.916.861.131.127.30361.251.224.050.970.946.2671.081.094.031.101.085.35370.790.784.911.010.985.6081.251.253.911.251.244.04380.850.844.450.980.956.7490.990.994.061.071.078.05391.061.086.070.930.966.84100.940.915.491.271.256.36401.141.132.910.960.986.33111.001.015.521.241.228.28410.970.996.690.960.9510.00121.141.175.271.201.205.21420.920.915.690.980.986.51130.930.914.801.191.206.15430.940.947.661.101.0510.50140.960.975.661.181.185.49441.081.055.820.981.004.82151.091.104.821.281.254.64450.940.915.781.241.264.94161.221.215.191.461.465.37460.870.874.541.401.415.97170.890.895.801.211.245.54470.920.925.151.291.294.88180.870.845.881.121.116.83481.091.093.601.021.009.19191.091.093.761.191.177.14490.880.855.171.081.125.40201.281.274.161.051.105.90500.960.965.241.031.064.60210.890.885.701.041.044.40510.930.935.160.971.026.27220.920.905.341.091.045.45521.031.016.161.051.053.77231.161.154.991.031.056.50531.051.036.540.970.985.22241.291.245.510.990.986.20540.980.975.851.041.023.79250.920.897.211.141.125.35551.071.064.321.010.995.18260.870.885.411.031.025.74561.021.055.691.081.056.77271.071.064.831.151.125.55570.960.964.391.091.115.08281.131.134.491.111.137.67581.021.026.680.890.906.72290.940.967.621.111.144.44591.081.084.891.031.085.24300.860.836.671.191.165.35601.091.096.701.151.198.99

Analysis of disc hold-release times

The time, muscle contraction levels, and the movement classification results obtained from them were recorded for the entire test. Hold-release times were obtained by subtracting the samples with values marked as hold as a result of motion classification from the samples obtained in the whole test. The results of the evaluation of EMG signals and motion classification for each participant during the placing and turning are given in Table 4. The times were determined by averaging 3 repetitions in each phase. Table 4. Comparison of disc hold times in placing and turning phasesPlacingTurningNoTotal time (s)Hold time (s)%Total time (s)Hold time (s)%P16228.345.66630.946.8P26527.141.76929.743.0P36334.755.16732.147.9P45725.444.66628.443.0P56129.848.97629.338.6P65519.735.85827.347.1P75626.547.36233.554.0P86226.142.16729.944.6P9603253.36427.142.3P105831.754.76724.336.3P115927.947.36230.148.5P125925.142.56725.738.4P136724.837.07726.133.9P146424.938.96935.251.0P156319.631.16929.342.5P166324.438.77030.343.3P176836.954.37126.737.6P186027.345.56428.744.8P196623.836.16723.134.5P20653350.87134.849.0Mean61.727.544.667.529.143.4Max6836.955.17735.254Min5519.631.15823.133.9Std. dev3.64.574.53.2****5.5

When Table 4 is examined, it is seen that the average time taken by the participant while the disc was in his/her hand during the placing phase conducted with 20 participants was 27.5 s. This result corresponds to 44.6% of the total average time. Considering the nature of the test, in the placing, the participant reaches out his/her hand, grabs the disc, and places it in its place, and the disc remains in his/her hand for almost half of the total time. Therefore, the result is compatible with the structure of the test. In the turning phase, the participant picks up the disc with one hand, turns it, and places it in place by passing it to the other hand. In each row of the test board, he changed the hand with which he grasped and turned the disc. When the test results were analyzed, it was found that the average time taken during the turning phase was 29.1 s. This result corresponds to 43.4% of the total average turning time. When the nature of the test is examined, the discs were grasped for most of the process. However, the average of the times obtained was below 50%. This time was lower than expected because only the dominant hand to which the MYO armband device was connected was measured. It is thought that the time will increase to the expected value when the EMG device is connected to both arms.

Analysis of arm movements

Wearable sensors for non-invasive detection of limb movements that occur during various activities are widely used today. These sensors overcome many of the drawbacks of camera-based systems, which are considered the gold standard for motion analysis, allowing the application of motion in its natural environment [28–30]. Inertial sensors [30, 31] are widely used to objectively measure body kinematics during various activities in a way that does not restrict movement. In this study, arm movements during dexterity tests were measured with the IMU sensor to detect possible movement limitations of individuals during the test and to suggest the diseases that should be focused on accordingly to the experts. With gyroscope data, roll, pitch, and yaw values were obtained. Table 5 shows the maximum angular velocity values for the placing and turning phases of the test. Table 5. Angular velocities of the arm during the disc placing and turning phaseRoll (°/s)Pitch (°/s)Yaw (°/s)NoPlacingTurningPlacingTurningPlacingTurningP194.5117.51049119340.5P2153132133.586.5222.568P370.590132.572.518345P4135.5102.5170.568.517796P5140.5130114.558159.584.5P6176.5164261.575.5177.5155P711989166.5106239.551P8123130113.5118211.552.5P911181.5888017235.5P10131120.51618719959P11136.596.51649922948P12124.512615393.519150.5P13166120106.559.5171.547.5P14148.5115159.595.520346P1513191.5118115.5163.581P16136.592.5123116200.581P17109.5106.516779.5175.560P18130120.5103.591170.562.5P19110.592.515266.5171.547P201258312293.522354.5Mean128.6110140.787.6191.663.3Max176.5164261.5118239.5155Min70.581.58858159.535.5Std. dev23.72138.317.923.5****26.9

When the data in Table 5 are examined, the average angular velocities in the positive and negative directions in the roll, pitch, and yaw axes for the placing phase were 125, 122, and 93.5, respectively, while these values were 83, 93.5, and 54.5 in the turning phase. When the angular velocity results for arm movements are analyzed, it is seen that the arm moves at smaller speeds in the turning phase. This is an expected result since the discs are rotated in the position they are in during the turning, the fingers are mostly moving, and no reaching movement occurs.

Table 6 shows the linear accelerations during the placing and turning phases. Average accelerations were determined by averaging the peak values for each direction. When the data in the table are examined, the average accelerations in the X, Y, and Z axes for the placing were 2.1, 2.7, and 1.3, respectively, while these values were 0.9, 1.2, and 0.8 in the turning phase. When the linear accelerations of the arm in the placing and turning were examined in general, it was observed that the values were close to each other and the standard deviation was low. This means that these values can be used as a reference for the movements of healthy individuals during the dexterity test and inferences can be made on the test results of the patients. The motion analysis results of healthy individuals are suitable for the reference values in the expert system rule base. In the future, more accurate reference values can be obtained by increasing the number of healthy participants from 20 and expanding the database. Table 6. Average linear accelerations of the arm during the disc placing and turning phasesX axis (m/s^2^)Y axis (m/s^2^)Z axis (m/s^2^)NoPlacingTurningPlacingTurningPlacingTurningP12.350.82.51.251.050.75P22.051.13.10.91.41.05P32.251.151.750.71.10.8P42.41.153.40.91.20.65P52.950.751.40.90.950.65P62.51.54.451.951.751.05P71.650.92.51.051.30.75P81.30.752.50.81.20.8P91.350.953.31.410.9P1021.252.91.451.951.3P112.450.81.851.21.10.85P121.750.653.451.051.350.65P131.750.72.70.91.10.6P141.650.851.950.651.20.65P152.713.351.251.350.7P162.250.752.351.551.40.75P1720.952.951.351.150.8P181.90.652.351.81.50.95P191.950.71.90.951.450.7P201.851.12.711.20.9Mean2.10.92.71.21.30.8Max2.91.54.51.91.91.3Min1.30.71.40.60.90.6Std. dev0.430.230.730.350.25****0.17

Conclusions

In this study, a system that can automatically perform hand dexterity test with a multi-sensor structure consisting of EMG, IMU, camera, and artificial neural network-based motion classification algorithm has been developed. In the system, the muscle contraction levels occurring during the hand dexterity test are taken from the MYO armband placed on the forearm and evaluated with the artificial neural network-based LLGMN algorithm, and the hold-release motion is classified. Angular velocity and linear acceleration values obtained from the 9-axis IMU sensor on the same device were used for arm motion analysis. Disc movements were tracked with a camera placed on the test table, and the placing and turning times of each disc were determined. The data was evaluated using a rule base created with the information received from experts, and inferences were made about the condition of the individuals. Thanks to this system, it will be easier to meet the demands, especially in populated areas. Human measurement errors will be minimized. By measuring durations and movement dynamics that cannot be measured by humans, it will ensure that the subcomponents of skill tests can also be used in diagnosis and treatment stages.

There are several limitations in the study. Shadows on the test board, lack of ambient light, and light direction negatively affect the image processing performance. To solve this problem, various changes are planned to be made in the position of the camera and the table on which the test board is placed. In the system, images are processed in Python program on Raspberry Pi, and EMG and motion data are processed in MATLAB program on the computer. By organizing all these data processing stages to run on a single controller, a compact embedded system will be obtained with less hardware. Another limitation of the system is that it is not possible to detect whether users have dropped discs. For this, it is planned to provide a solution based on image processing or with a sensor to be placed on the discs.