Scaling laws in wearable human activity recognition

TL;DR

This paper establishes the first scaling laws for wearable human activity recognition, showing how model performance improves with data and model size, emphasizing the importance of data diversity.

Contribution

It introduces the first known scaling laws for HAR, linking model capacity and data volume, and highlights the significance of data diversity for model performance.

Findings

Pre-training loss follows a power law with data and parameter count.

Increasing user diversity improves performance more than increasing data per user.

Scaling laws improve downstream HAR task performance on benchmark datasets.

Abstract

Many deep architectures and self-supervised pre-training techniques have been proposed for human activity recognition (HAR) from wearable multimodal sensors. Scaling laws have the potential to help move towards more principled design by linking model capacity with pre-training data volume. Yet, scaling laws have not been established for HAR to the same extent as in language and vision. By conducting an exhaustive grid search on both amount of pre-training data and Transformer architectures, we establish the first known scaling laws for HAR. We show that pre-training loss scales with a power law relationship to amount of data and parameter count and that increasing the number of users in a dataset results in a steeper improvement in performance than increasing data per user, indicating that diversity of pre-training data is important, which contrasts to some previously reported findings in self-supervised HAR. We show that these scaling laws translate to downstream performance improvements on three HAR benchmark datasets of postures, modes of locomotion and activities of daily living: UCI HAR and WISDM Phone and WISDM Watch. Finally, we suggest some previously published works should be revisited in light of these scaling laws with more adequate model capacities.