Federated Action Recognition for Smart Worker Assistance Using FastPose

TL;DR

This paper introduces a federated learning framework for skeleton-based human activity recognition in industrial environments, improving accuracy and privacy without centralized data collection.

Contribution

It proposes a novel federated learning approach using FastPose and evaluates its effectiveness with new industrial gesture datasets and models.

Findings

Federated learning significantly improves cross-user generalization.

FedEnsemble outperforms centralized training by +16.3 percentage points.

FL and FedEnsemble achieve higher accuracy on unseen clients.

Abstract

In smart manufacturing environments, accurate and real-time recognition of worker actions is essential for productivity, safety, and human-machine collaboration. While skeleton-based human activity recognition (HAR) offers robustness to lighting, viewpoint, and background variations, most existing approaches rely on centralized datasets, which are impractical in privacy-sensitive industrial scenarios. This paper presents a federated learning (FL) framework for pose-based HAR using a custom skeletal dataset of eight industrially relevant upper-body gestures, captured from five participants and processed using a modified FastPose model. Two temporal backbones, an LSTM and a Transformer encoder, are trained and evaluated under four paradigms: centralized, local (per-client), FL with weighted federated averaging (FedAvg), and federated ensemble learning (FedEnsemble). On the global test set, the FL Transformer improves over centralized training by +12.4 percentage points, with FedEnsemble delivering a +16.3 percentage points gain. On an unseen external client, FL and FedEnsemble exceed centralized accuracy by +52.6 and +58.3 percentage points, respectively. These results demonstrate that FL not only preserves privacy but also substantially enhances cross-user generalization, establishing it as a practical solution for scalable, privacy-aware HAR in heterogeneous industrial settings.