HRTR: A Single-stage Transformer for Fine-grained Sub-second Action Segmentation in Stroke Rehabilitation

TL;DR

This paper introduces HRTR, a single-stage transformer model designed for precise, sub-second action segmentation in stroke rehabilitation, outperforming existing methods without additional refinements.

Contribution

The work presents a novel high-resolution temporal transformer that directly localizes and classifies fine-grained, sub-second actions in a single stage, simplifying the process.

Findings

HRTR achieves higher Edit Scores than state-of-the-art methods.

HRTR performs well on both stroke rehabilitation and general datasets.

The model eliminates the need for multi-stage processing and post-processing.

Abstract

Stroke rehabilitation often demands precise tracking of patient movements to monitor progress, with complexities of rehabilitation exercises presenting two critical challenges: fine-grained and sub-second (under one-second) action detection. In this work, we propose the High Resolution Temporal Transformer (HRTR), to time-localize and classify high-resolution (fine-grained), sub-second actions in a single-stage transformer, eliminating the need for multi-stage methods and post-processing. Without any refinements, HRTR outperforms state-of-the-art systems on both stroke related and general datasets, achieving Edit Score (ES) of 70.1 on StrokeRehab Video, 69.4 on StrokeRehab IMU, and 88.4 on 50Salads.