HAROOD: A Benchmark for Out-of-distribution Generalization in Sensor-based Human Activity Recognition

TL;DR

HAROOD is a comprehensive benchmark for evaluating out-of-distribution generalization in sensor-based human activity recognition across multiple scenarios, datasets, and algorithms, highlighting the need for further research.

Contribution

The paper introduces HAROOD, a new benchmark with diverse OOD scenarios, datasets, and methods, providing a standardized platform for future HAR research under distribution shifts.

Findings

No single method consistently outperforms others across scenarios.

Substantial opportunities exist for developing more effective OOD algorithms for HAR.

The benchmark facilitates systematic evaluation and comparison of OOD methods in HAR.

Abstract

Sensor-based human activity recognition (HAR) mines activity patterns from the time-series sensory data. In realistic scenarios, variations across individuals, devices, environments, and time introduce significant distributional shifts for the same activities. Recent efforts attempt to solve this challenge by applying or adapting existing out-of-distribution (OOD) algorithms, but only in certain distribution shift scenarios (e.g., cross-device or cross-position), lacking comprehensive insights on the effectiveness of these algorithms. For instance, is OOD necessary to HAR? Which OOD algorithm performs the best? In this paper, we fill this gap by proposing HAROOD, a comprehensive benchmark for HAR in OOD settings. We define 4 OOD scenarios: cross-person, cross-position, cross-dataset, and cross-time, and build a testbed covering 6 datasets, 16 comparative methods (implemented with CNN-based and Transformer-based architectures), and two model selection protocols. Then, we conduct extensive experiments and present several findings for future research, e.g., no single method consistently outperforms others, highlighting substantial opportunity for advancement. Our codebase is highly modular and easy to extend for new datasets, algorithms, comparisons, and analysis, with the hope to facilitate the research in OOD-based HAR. Our implementation is released and can be found at https://github.com/AIFrontierLab/HAROOD.