Weakly Supervised Recognition of Surgical Gestures

TL;DR

This paper presents a weakly supervised approach for recognizing surgical gestures from robotic kinematic data, leveraging minimal expert input to improve unsupervised segmentation accuracy and reduce manual annotation efforts.

Contribution

The study introduces a GMM-based gesture recognition method initialized with minimal expert demonstrations, outperforming standard methods and enhancing accuracy through action redefinition and input optimization.

Findings

Significant improvement over standard initialization methods.

Effective use of minimal expert demonstrations.

Enhanced recognition accuracy with input optimization.

Abstract

Kinematic trajectories recorded from surgical robots contain information about surgical gestures and potentially encode cues about surgeon's skill levels. Automatic segmentation of these trajectories into meaningful action units could help to develop new metrics for surgical skill assessment as well as to simplify surgical automation. State-of-the-art methods for action recognition relied on manual labelling of large datasets, which is time consuming and error prone. Unsupervised methods have been developed to overcome these limitations. However, they often rely on tedious parameter tuning and perform less well than supervised approaches, especially on data with high variability such as surgical trajectories. Hence, the potential of weak supervision could be to improve unsupervised learning while avoiding manual annotation of large datasets. In this paper, we used at a minimum one expert demonstration and its ground truth annotations to generate an appropriate initialization for a GMM-based algorithm for gesture recognition. We showed on real surgical demonstrations that the latter significantly outperforms standard task-agnostic initialization methods. We also demonstrated how to improve the recognition accuracy further by redefining the actions and optimising the inputs.

Figures

Tables

Table 1. TABLE I: Kinematic feature vector.

Indices	Description of variables
1–3	Right PSM tool tip position ($xyz$)
4–7	Right PSM tool tip orientation quaternion (q)
8–10	Right PSM tool tip linear velocity ($\dot{x}\dot{y}\dot{z}$)
11–13	Right PSM tool tip rotational velocity ($\dot{\alpha }\dot{\beta }\dot{\gamma }$)
14	Right PSM gripper angle ($\theta$)
15–17	Left PSM tool tip position ($xyz$)
18–21	Left PSM tool tip orientation quaternion (q)
22–24	Left PSM tool tip linear velocity ($\dot{x}\dot{y}\dot{z}$)
25–27	Left PSM tool tip rotational velocity ($\dot{\alpha }\dot{\beta }\dot{\gamma }$)
28	Left PSM gripper angle ($\theta$)
29–32	Euclidean distance signals ($d_x,d_y,d_z,d$)

Table 2. TABLE II: Results on Expert demonstrations.

Annotations		Original	Proposed
Initialization		${\text{GMM}}_{\mathrm{𝟎}}$	K-means	${\text{GMM}}_{\mathrm{𝟎}}$
Signals		All			No pose	No velocity	No distance
	$𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚$	58%	\	83%	55%	85%	77%
Extrinsic metrics	$𝑵𝑴𝑰$	0.58	0.58	0.72	0.41	0.73	0.67
	$𝑺{𝑰}_{𝑮𝑴𝑴\mathrm{𝟎}}$	0.55	0.52	0.57	0.48	0.60	0.55
Intrinsic metrics	$𝑺{𝑰}_{𝑮𝑻}$	0.59	0.56	0.56	0.54	0.56	0.56

Table 3. TABLE III: Results on extended dataset.

Metrics		E	E+I	E+I+N
	$𝑨𝒄𝒄𝒖𝒓𝒂𝒄𝒚$	85%	77%	59%
Extrinsic metrics	$𝑵𝑴𝑰$	0.73	0.66	0.46
	$𝑺{𝑰}_{𝑮𝑴𝑴\mathrm{𝟎}}$	0.60	0.53	0.50
Intrinsic metrics	$𝑺{𝑰}_{𝑮𝑻}$	0.54	0.54	0.52

Equations

$$\text{x}(t+1)={\text{A}}_k\text{x}(t)+\text{w}(t),{\text{A}}_k\in \{{\text{A}}_1,\cdots ,{\text{A}}_N\}$$

$$\text{n}(t)=[\text{x}(t),\text{x}(t+1),\cdots \text{x}(t+W)]\in R^{pW}.$$

$$\textbf{e}_{1}(t),\textbf{e}_{2}(t),\textbf{e}_{3}(t)\ \xrightarrow{}\ \{\mbox{\boldmath$\mu$\unboldmath${}_{0k}$},\ \textbf{C}_{0k},\ w_{0k}\}_{k=1:N}$$

$$NMI(X,Y)=\frac{I(X,Y)}{\sqrt{H(X)H(Y)}}$$

$$s(i)=\frac{b(i)-a(i)}{max\{a(i),b(i)\}}$$

$$SI=\sum _{i=1}^{Ns}s(i)$$

Full text

I Introduction

Robot-Assisted Minimally Invasive Surgery (RAMIS) is an established practice across a range of surgical specialties, which helps to improve precision of the surgical manipulation and ergonomic comfort of the surgeon [1]. With RAMIS, a large dataset of video and kinematic trajectories of surgical interventions can be recorded from the robotic system, e.g. da Vinci surgical system (dVSS, Intuitive Surgical Inc., CA, USA). Surgical gesture recognition, i.e. segmentation and labelling of surgical action units, by analysing these datasets automatically can be used for multiple purposes, e.g. surgical skills assessment [2, 3] and automation [4, 5].

However, automatic gesture recognition is difficult to implement robustly due to the complexity of surgical tasks and the variability in users’ actions and patient-specific anatomy [7]. A number of approaches have been proposed to address this problem. Classical approaches are based on statistical models such as Gaussian Mixture Models (GMM) [8], Hidden Markov Models (HMM) [9, 10] and Conditional Random Fields (CRF) [11, 12]. More recently, deep learning techniques have also been employed, providing the current state-of-the-art results [13, 14, 15]. The challenge with most of these methods is that they use manual annotations, which are costly, time consuming and subjective when generated by multiple participants. Due to subjectivity and smooth transitions between gestures, boundaries between consecutive gestures are often not clearly defined. Unsupervised methods, that automatically learn the segmentation criterion from the data, have been developed to overcome these limitations [8], [16], [17]. However, they typically perform less well than methods trained on labelled information, especially on data with high variability such as surgical trajectories. Hence, the potential of weak supervision could be to improve unsupervised learning while avoiding manual annotation of large datasets.

The aim of this paper is to propose a new weakly supervised approach for surgical gesture recognition, that allows to retain the amount of annotations of surgical gestures limited to very few demonstrations. In particular we used at a minimum one expert demonstration and its ground truth annotations to generate an appropriate initialization for a GMM-based unsupervised recognition algorithm, in order to improve upon standard task-agnostic initialization methods, such as random or K-means initialization [18]. We focused on recognition of surgeme units (Fig.1), the shortest “surgical motion unit with explicit semantic sense” [16] (e.g. grasping the needle, pulling the suture, etc.). We validated our algorithm on the JIGSAWS dataset [6, 19], featuring suturing demonstrations collected from eight surgeons with different skill level using the dVSS.

II Related work

GMM-based methods: this group of works segment robot trajectories into action classes by fitting a GMM onto the available samples. In [20] the number of mixture components was chosen using the Bayesian Information Criterion (BIC) and the fitting was initialized using the K-means clustering algorithm. [8] presented a multi-level clustering approach for identification and pruning of segmentation points, which is based on a Dirichlet Process GMM (DPGMM), i.e. a mixture model where the number of clusters is determined by a DP. This work was extended integrating features extracted with deep neural networks from the video data [21], improving the recognition accuracy.

Heuristic initialization: a number of studies make use of heuristics to create an initial segmentation of the data. Examples include heuristics based on Zero Crossing Velocity [22], jerk profiles [23] or trajectory curvature [24]. However, the same heuristic could be not appropriate to explain every part of the data, and the output is often over-segmented [24].

Weakly supervised methods: only few weakly supervised approaches have been developed for surgical action recognition [25], [26], [27]. Some works, however, assume that the actions follow a pre-defined order, the goal is to find the action boundaries [26], [27]. In addition, these methods were only applied to recognise surgical phases, which represent high-level surgical states, using only the video data and lack the recognition of low-level surgeme units.

III Methods

Our approach relies on classical GMM clustering. Multiple reasons make GMM an appealing method for gesture recognition, such as performing simultaneous segmentation and classification, where one task does not rigidly influence the other, as in sequential approaches. GMM is intuitive because action classes are represented through independent means, covariance matrices and weights. Finally, the fuzziness at the segment boundaries is modelled through Gaussian intersections.

Notation: vectors are represented in bold lowercase letters (e.g. x), matrices are represented in bold capital letters (e.g. A) and scalars are represented in italic letters (e.g. t).

III-A Data pre-processing

We used JIGSAWS [6], a public dataset comprising video and kinematic data captured at 30 Hz from the dVSS during multiple demonstrations of elementary surgical tasks, which were performed on phantoms by eight surgeons with different robotic surgical experience (expert, intermediate, novice). JIGSAWS also contains manual annotations describing the ground truth segmentation of each demonstration into action classes.

We tested our algorithm on the kinematic data recorded from the two Patient Side Manipulators (PSM1 and PSM2) of the dVSS [28]. The motion of each arm is described by a local frame attached at its end-effector using 19 kinematic variables, including Cartesian positions, a rotation matrix, linear velocities, angular velocities and a gripper angle.

The pre-processing pipeline of [16] was implemented:

• •

The rotation matrix R describing the end-effector orientation with respect to the robot base is converted into a more compact quaternion representation q, reducing the state vector to 14 variables for each arm.

• •

All the trajectories are smoothed with a low-pass filter with cut-off frequency $f_{c}=1.5$ Hz in order to minimize the measurement noise.

• •

All the trajectories are normalized to zero mean and unit variance, in order to enable a fair comparison between signals with different unit of measure.

Additionally, four signals representing the distance between the two end-effectors along the three orthogonal axes ($d_{x},d_{y},d_{z}$) and their absolute Euclidean distance ($d$) are generated, in order to include information about the relationship between the two manipulators, resulting in a state vector $\textbf{x}(t)\in R^{p}$ of $p=32$ variables. Table I describes in detail the variables included in the kinematic feature vector.

Finally, the trajectories are subsampled from 30 Hz to 10 Hz for faster computation time.

III-B Simultaneous action segmentation and recognition

We build upon the approach of [8], treating each demonstration $\textbf{x}(t)\in R^{p}$, as a realization of a switched linear dynamical system with zero-mean Gaussian process noise $\textbf{w}(t)\in R^{p}$:

[TABLE]

where each different locally linear regime $\textbf{A}_{k}\in R^{p\times p}$ corresponds to one of the $N$ different surgemes composing the task. As explained in [8], under this hypothesis action recognition can be performed by fitting a GMM to the augmented state $\textbf{n}(t)$, defined as:

[TABLE]

When $W=1$, GMM fitting is indeed equivalent to solving multiple linear regression problems [29], one for each action class $\textbf{A}_{k}$. After model fitting, each trajectory sample is assigned to its most likely mixture component, i.e. its most likely surgeme label.

III-C Weakly supervised initialization

In order to initialize the GMM parameters (mean, covariance and weight of each mixture component), we use a small set of manually-segmented demonstrations. This set is composed of two demonstrations from expert users ($\textbf{e}_{1}(t),\textbf{e}_{2}(t)$) and one demonstration from an intermediate user ($\textbf{e}_{3}(t)$), randomly selected among all demonstrations observed to be free from execution errors (such as needle dropping or multiple attempts of the same gesture). This set provides an exemplary execution of each possible action, which helps to generate a mixture with the correct number of components and appropriate shape. Exploiting the available ground truth annotations, we fit an initial GMM, denoted as $GMM_{0}$, to the example demonstrations ($\textbf{e}_{1}(t),\textbf{e}_{2}(t),\textbf{e}_{3}(t)$), thus obtaining the initial values of the mean vector ($\mu$0k), covariance matrix ($\textbf{C}_{0k}$) and weight ($w_{0k}$) of each mixture component:

[TABLE]

III-D Trajectory segmentation

Once the initial mixture parameters have been generated, offline segmentation of the full dataset was performed by fitting another GMM onto the unlabelled demonstrations using the Expectation Maximization (EM) algorithm [30]. Each trajectory sample is assigned to its most likely mixture component, i.e. its most likely action label.

III-E Ground Truth segmentation redefinition

As introduced in Section B, the action recognition method of this study relies on the hypothesis of local linearity in demonstrations, where each locally linear regime corresponds to a different surgeme. Thus, we decided to analyse the suturing task in order to check if this hypothesis is approximately verified.

When observing the video records of the suturing demonstrations, we noticed the presence of sudden motion variations during the execution of some of the surgemes. By plotting (Fig. 2 top) the corresponding kinematic state vector $\textbf{n}(t)$ as an image (where each pixel row corresponds to the value, mapped into gray levels, of a kinematic feature in time), overlaid with ground truth action boundaries (labels G), we noticed indeed the presence of sharp transitions of the kinematic pattern within those surgemes (e.g. see G3 and G6). In order to better satisfy the local linearity hypothesis required by our recognition algorithm, we therefore redefined our ground truth annotations (Fig. 2 bottom) in a way to avoid abrupt motion variations within surgemes.

Using the video feedback and the original annotations, all the trajectories have therefore been re-segmented according to the following criteria:

• •

Surgeme (G3) pushing needle through the tissue is split into (L1) pushing needle through tissue and (L2) transferring needle from right to left.

• •

Surgeme (G6) pulling suture with left hand is split into (L5) extracting suture from tissue with left hand and (L3) pulling suture with left hand.

• •

Surgeme (G11) dropping suture and moving to end points is, when necessary, split into (L7) orienting needle, (L9) dropping suture, and (L10) moving to end points.

• •

Surgeme (G5) moving to centre of workspace with needle in grip is most of the times performed simultaneously with (L7) orienting needle or (G2) positioning the tip of the needle. We therefore include it in either of the two.

• •

Surgeme (G2) positioning the tip of the needle and (G3) pushing needle through the tissue are merged into (L1) pushing needle through tissue class, because there is no clear transition point between the two actions. Moreover, small needle repositioning motions are often performed when inserting the needle through the tissue.

The redefined action dictionary is presented in Fig. 3.

IV Evaluation metrics

We evaluate our algorithm performance in a similar way to [21]. We use both extrinsic metrics, comparing the segmentation result to the ground truth annotations, and intrinsic metrics, measuring the compactness of the generated transition point clusters.

IV-A Extrinsic metrics

Accuracy: The accuracy represents the percentage of correctly labelled frames.   Normalized Mutual Information (NMI): The NMI measures the alignment between two sequences of labels (X and Y):

[TABLE]

where $I$ is the mutual information and $H$ the entropy [31]. This metric is independent of the absolute values of the labels, i.e. the score is not affected by permutations of cluster labels.

IV-B Intrinsic metrics

Silhouette Index (SI): The SI for a single sample is defined as:

[TABLE]

where $a(i)$ is the distance between that sample and the mean of the cluster it belongs to, while $b(i)$ is the distance between that sample and the mean of the nearest cluster it is not part of. We employed the Euclidean distance metric. The Silhouette value is a measure of how similar an object is to its own cluster compared to other clusters. The individual Silhouette value ranges from -1 to +1. We normalised it between 0 and 1, as in [21]. A high value indicates that the object is well matched to its own cluster and poorly matched to neighbouring clusters. $SI$ is the average Silhouette score over all Ns samples:

[TABLE]

High $SI$ indicates that the clustering configuration is appropriate. We call $SI_{GMM0}$ the SI computed on the clusters identified by our algorithm, and $SI_{GT}$ the SI computed on the Ground Truth clusters.

V Experiments and Results

As described in Fig. 4, we conducted two sets of experiments, the first on a dataset comprising only expert demonstrations, and the second on a dataset comprising expert, intermediate and novice demonstrations.

JIGSAWS features 10 suturing demonstrations from expert surgeons, 10 suturing demonstrations from intermediate surgeons and other 20 from novice surgeons. Each demonstration has a different duration of approximately $1105\pm 432$ frames. The expert demonstrations have generally the shortest duration.

V-A Sliding window length

First of all, we conducted a preliminary test to select the optimal sliding window length (W), which defines the dimensions of the augmented feature state $\textbf{n}(t)$. We applied our $GMM_{0}$ method on expert demonstrations for increasing values of W, and computed the corresponding accuracy score with redefined dictionary. As illustrated in Fig. 5, the trend is negative, as accuracy decreases when W increases. This could be explained by the corresponding increase in feature dimensionality, which jeopardises the clustering robustness given the same amount of samples. We selected W=2, providing the best recognition performance.

V-B Ground truth redefinition

We then analysed the performance of our $GMM_{0}$ method on both the original and the proposed ground truth annotations. The results are summarized in Table II. With the proposed annotations the extrinsic metrics show remarkable improvement with respect to the original ground truth, with accuracy score increasing of 25% and NMI score increasing of 14%. The $SI_{GMM0}$ is also improved, advancing from 0.55 to 0.57.

In addition, we compared the recognition accuracy between groups of corresponding labels belonging to the original and the proposed action dictionaries. As shown in Fig. 6, the fusion of surgeme G2 and G3 into L1 and the separation of surgeme G3 into L1 and L2 give rise to action classes L1 and L2 which can be recognized more robustly, while the separation of G6 into L5 and L3 does not generate significant variations. Recognition accuracy of G11, split into L9 and L10, decreases with the proposed annotations, but the accuracy of all the other labels (G1, G4, G9, G8) is mostly improved. Overall, a more robust GMM distribution is generated when initialized with the proposed action dictionary. These results underline the influence of the action dictionary definition on action recognition performance.

V-C Initialization technique

We compared the performance of our $GMM_{0}$ initialization method with respect to the commonly used K-means initialization method [32], with redefined dictionary (see Table II). The number of K-means clusters is set as the number of action labels in the dictionary and the initial seeds are randomly sampled from the dataset. Being K-means algorithm fully unsupervised, no information about the identity of the generated clusters is available. For this reason the accuracy score was not computed.

$GMM_{0}$ initialization leads to 14% improvement of NMI, as well as increase of $SI_{GMM0}$. K-means indeed assumes equal prior probability for all K clusters (i.e. each cluster has roughly the same number of observations) [33] and it is randomly initialized. $GMM_{0}$, on the other hand, exploits prior information to model the initial location, shape and size of the clusters, leading to more robust action identification.

V-D Feature selection

We also studied the influence of different signals on the recognition accuracy. Specifically, we analysed the contribution of the pose (NO pose), velocity (NO velocity) and Euclidean distance (NO distance) signals by observing how the performance changes in their absence. The results in Table II suggest that the velocity signals should be discarded, as the recognition performance improves when they are excluded. Velocity signals are indeed a major source of within-cluster variability: not only users with different expertise level perform surgical tasks at different speeds, but even within the same demonstration velocity signals belonging to the same action class show high variability (see Fig. 7). The Euclidean distance signals we introduced, instead, give positive contribution to the classification accuracy, as the recognition performance degrades when they are excluded.

Fig. 8 shows an example of segmentation output, compared to its ground truth. Fig. 9 shows the 2D distribution, obtained with the t-SNE visualization technique [34], of the transition points identified by our algorithm on the expert set. Cluster compactness is visually comparable to the ground truth distribution.

V-E Extended dataset

Finally we extended our algorithm validation to the full dataset, in order to test the robustness of our method, with redesigned dictionary and selected features, against increasing data variability and the presence of spurious motions. Specifically, we used the same initialization ($GMM_{0}$) as in the previous experiments, but we extended the unsupervised GMM fitting to all expert (E), intermediate (I) and novice (N) demonstrations. As summarized in Table III, the lower the expertise level of the surgeon, the lower the final accuracy, NMI and $SI_{GMM0}$ scores. Simple GMM approaches do not exploit temporal constraints such as transition probabilities between actions. This constitutes a major limitation in the analysis of sequential information such as kinematic trajectories, resulting in limited performance as the data variability increases.

VI Conclusion

This paper explored a new weakly supervised approach for surgical gesture recognition, that allows to retain the amount of annotations limited to very few demonstrations. We employed three demonstrations and their ground truth annotations to generate an appropriate initialization for a GMM-based recognition algorithm. Experimental results on real surgical kinematic trajectories during a training exercise confirm that weakly supervised initialization significantly outperforms standard task-agnostic initialization methods. We also demonstrated that recognition accuracy can be improved by carefully designing the optimal channel selection and the appropriate action granularity for the specific task at hand. We believe that inclusion of contextual and semantic information [35, 36] from video data would further boost the recognition performance [21].

However, manual redefinition of the action dictionary based on visual verification still involves a certain degree of subjectivity, and further validation should also be performed to assess the recognition performance for different sets of manually-segmented demonstrations employed in the initialization step. In addition, simple GMM approaches are not robust against increasing data variability. More complex GMM-based methods have been developed specifically for time series analysis, such as Gaussian-HMM [37] and GMM-HMM [38, 39]. These models introduce transition probabilities between different actions, thus generating a probabilistic action grammar that helps to improve the recognition accuracy. In future work, we will explore the effects of weak supervision on the initialization of transition and observation probability distributions in unsupervised HMM-based approaches. Finally, our experiments were conducted on a small-scale dataset, raising concerns about the generalization capability of our approach to surgical data modelling more broadly. More in-depth analysis will be performed on larger and more challenging datasets of robotic surgical demonstrations, e.g. [40].