# Action recognition in medical environments for robotic assistance

## TL;DR

This paper explores using action recognition to enable robotic assistance during handovers in medical settings, such as surgeries and patient care.

## Contribution

A novel framework for robotic assistance in medical handovers using skeletal data and machine learning for action recognition.

## Key findings

- The framework achieved an F1 score of 0.736 for surgical handovers and 0.941 for care interventions.
- Actions with rapid transitions, like approach and reach, were more challenging to distinguish.
- The method shows potential for enabling robots to assist in medical material handovers.

## Abstract

Teamwork is fundamental to medical practice and relies on seamless collaboration among professionals with different tasks. Integrating robotic systems into this environment demands smooth interactions. Human action recognition, which infers a person’s state without explicit input, can support this. We focus on handovers between medical staff, using the actions as implicit cues for robotic assistance to replace the giving party in such scenarios.

Skeletal information processed with differing machine learning algorithms makes it possible to derive actions out of sequential image data. Transferred to the medical context, we aim to infer actions defined for each situation in two datasets, a surgery in the operating room and a care intervention in the patient ward, depicting a handover between staff. We aim to abstract movement patterns across individuals through skeletal representation, leveraging the spatiotemporal information of medical handovers to enable future robotic systems to interact based on implicit cues.

We report an F1 score of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.736 \pm 0.045$$\end{document}0.736±0.045 for the OR dataset with ST-GCN and an F1 score of \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$0.941 \pm 0.009$$\end{document}0.941±0.009 for the Ward dataset with the SkateFormer human action recognition. The defined actions showed distinction in the confusion matrix with limitations on actions with a rapid transition like approach and reach as well as the handover actions in the OR.

The handover phases in two medical contexts, a minimally invasive surgery and a wound dressing on the patient station, are recognized with the proposed framework. This lays a first step for the integration of robotic assistance in the handover of medical material or instruments.