MMTA: Multi Membership Temporal Attention for Fine-Grained Stroke Rehabilitation Assessment

TL;DR

This paper introduces MMTA, a novel high-resolution temporal transformer that captures fine-grained micro-movements in rehabilitation videos and IMU data, improving assessment accuracy without added complexity.

Contribution

We propose Multi-Membership Temporal Attention (MMTA), a new approach that enhances boundary sensitivity in temporal segmentation by allowing frames to attend to multiple local contexts simultaneously.

Findings

MMTA improves Edit Score by +1.3 on StrokeRehab and +3.3 on 50Salads.

It supports both video and IMU inputs in a single-stage architecture.

Ablation studies confirm performance gains are due to multi-membership temporal views.

Abstract

To empower the iterative assessments involved during a person's rehabilitation, automated assessment of a person's abilities during daily activities requires temporally precise segmentation of fine-grained actions in therapy videos. Existing temporal action segmentation (TAS) models struggle to capture sub-second micro-movements while retaining exercise context, blurring rapid phase transitions and limiting reliable downstream assessment of motor recovery. We introduce Multi-Membership Temporal Attention (MMTA), a high-resolution temporal transformer for fine-grained rehabilitation assessment. Unlike standard temporal attention, which assigns each frame a single attention context per layer, MMTA lets each frame attend to multiple locally normalized temporal attention windows within the same layer. We fuse these concurrent temporal views via feature-space overlap resolution, preserving competing local contexts near transitions while enabling longer-range reasoning through layer-wise propagation. This increases boundary sensitivity without additional depth or multi-stage refinement. MMTA supports both video and wearable IMU inputs within a unified single-stage architecture, making it applicable to both clinical and home settings. MMTA consistently improves over the Global Attention transformer, boosting Edit Score by +1.3 (Video) and +1.6 (IMU) on StrokeRehab while further improving 50Salads by +3.3. Ablations confirm that performance gains stem from multi-membership temporal views rather than architectural complexity, offering a practical solution for resource-constrained rehabilitation assessment.